XtreemFS

• Features & Use Cases
  • Features
  • Overview
  • Specific Use Cases
  • HPC cluster
  • Hadoop cluster
• Download
• Support
  • Documentation
  • Quick Start
  • User Guide
  • FAQs
  • Support
  • Mailing List
• Contribute
• About

Fault-tolerant File Replication

XtreemFS replicates your file data across multiple storage servers. Our replication algorithm is designed to automatically cope with all the problems and failure scenarios that can occur in truly distributed systems: message loss, network partitions, server crashes etc.

For each replicated file, XtreemFS keeps a separate list of replica. This gives you complete control where to place replicas. When a replicated file is opened, XtreemFS automatically elects a primary replica for that file which processes all file operations. If the primary fails, one of the backup replicas will automatically take over after a short failover period.

Your applications won't see a difference between regular and replicated file because the XtreemFS file replication is completeley transparent. In the video below you can see XtreemFS file replication and automatic failover in action. We used an unmodified mplayer showing a video stored on XtreemFS.

Read-only replication

Read-only replication distributes immutable files to a large number of replicas and helps you quickly build a caching infrastructure on top of XtreemFS. In addition to full replicas that contain a complete copy, XtreemFS also supports partial replicas. These replicas are filled on demand when a client accesses data.

---

Virtualized, Elastic & Scalable

You can quickly scale up or down an XtreemFS installation by adding or removing storage servers. An XtreemFS installation can scale to thousands of storage and metadata servers. The elasticity and scalability makes XtreemFS a perfect Cloud File System.

Cloud + VM Storage

As XtreemFS virtualizes the storage resources of your cluster, it is the ideal backend for cloud infrastructures. XtreemFS can host your VM images independent from the machine the VM is running on (and simplify migrations between hosts), and it provides file system volumes that you can mount inside your VMs and use as a common large-scale storage for shared data.

Extensible

We have designed XtreemFS so that you can easily customize it for your requirements. Functions such as storage server selection, replica placement or user authentication are controlled through policies. If you need more control than our policies offer, you can easily write your own policies and plugins.

---

Metadata Backups

With asynchronous MRC backups you can easily create a consistent snapshot of your metadata without interrupting operations. The MRC will create the backup in background and continue to process client requests.

POSIX Compatibility

XtreemFS has the same "Look&Feel" as NFS or your local file system. Applications and users won't see a difference, also for replicated and remote files. You can use XtreemFS as a drop-in replacement for NFS.

User Management

XtreemFS comes with built-in support for SSL and X.509 certificates and can be directly integrated into authentication infrastructures and Virtual Organizations (VOs). XtreemFS is secure, because authorization is done by servers.

---

Parallel IO / Striping

XtreemFS implements striping of files over several storage servers for increased I/O bandwidth and capacity. This is similar to RAID0. Parallel I/O ensures that read/write operations are executed in parallel on the storage servers to exploit the full link bandwidth.

Filesystem Snapshots

XtreemFS supports versioned files and asynchronous metadata snapshots. Together these two techniques allow you to create snapshots of a volume instantaneously.

Metadata Caching (client)

The XtreemFS client can cache metadata to speed up interactive use, e.g. from your shell. This feature is particularly useful if you mount volumes over the WAN or slow links like DSL.