One namespace, many disks

What a DFS provides

The core service is a shared namespace: a tree of directories and files that looks the same from every client, whatever disk actually holds the bytes. A client attaches part of a remote tree into its own by mounting it.

Beyond a shared tree, a good DFS aims for the transparency goals of any distributed system. Location transparency means paths don't reveal servers. Access transparency means remote files use the same open/read/write calls. Replication and failure transparency mean users don't see copies or crashes. It also has to scale to many clients and much data without one server becoming the limit.

Design choices

Every distributed file system answers the same few questions. The answers explain most of their differences.

Where do operations run?

Sharing semantics

When two clients use the same file, what does a reader see after a writer writes? On one machine the answer is simple: the latest write, immediately. Across a network, keeping that promise is expensive, so file systems offer weaker contracts.

Weaker semantics are not a defect. They are what lets caching work. An application that needs stronger guarantees adds them itself, typically with file locks or by writing to a new file and renaming it into place atomically.

NFS: stateless sharing

Sun's Network File System (1985) aimed for one thing above all: easy recovery. In versions 2 and 3, the server keeps no per-client state. Each request carries everything needed to serve it: a file handle identifying the file, an offset, and a length. If the server crashes and reboots, clients just retry and continue. There are no sessions to rebuild.

Statelessness pushes NFS toward idempotent operations. "Write these bytes at offset 8192" can safely be retried; "append these bytes" cannot. Clients cache data and attributes for a few seconds (configurable, typically 3 to 60). They flush dirty data on close and revalidate on open, which yields close-to-open consistency. Between those points, two clients can see different contents.

The cost of statelessness is that the server can't help with caching or locking, so those were bolted on as separate protocols. NFSv4 (2000s) gave up pure statelessness. It adds open state, integrated locking, leases for crash recovery, and delegations, which let the server hand a client temporary exclusive rights to cache a file aggressively.

Papers and specsSandberg et al., "Design and Implementation of the Sun Network Filesystem," USENIX Summer 1985 · RFC 7530 (NFSv4) · RFC 8881 (NFSv4.1, pNFS)
TutorialInstall and configure an NFS server (Ubuntu) · Linux NFS wiki

AFS: whole-file caching

The Andrew File System at Carnegie Mellon (mid-1980s) was built for thousands of workstations per campus. Measurements showed most files are small, read far more than written, and rarely shared for writing. So AFS copies the entire file to the client's local disk on open. All reads and writes happen locally, and changed files go back to the server on close.

Instead of clients asking "has this changed?", the server makes a callback promise: it records which clients cache each file and notifies them when another client stores a new version.

The result is session semantics, with the server as the record of who caches what. Because clients rarely contact the server, AFS reached roughly ten times more clients per server than NFS in its era. Callback state must be rebuilt after a crash, and clients must refresh callbacks after a network outage, which makes AFS more complex than NFS. AFS also introduced a single global namespace (/afs/cell/…), Kerberos authentication, and volumes that can move between servers without changing paths. Its successor, Coda, added disconnected operation: laptops keep working offline and reconcile changes later.

PapersHoward et al., "Scale and Performance in a Distributed File System," ACM TOCS 6(1), 1988 · Satyanarayanan, "Scalable, Secure, and Highly Available Distributed File Access," IEEE Computer 23(5), 1990 · Kistler & Satyanarayanan, "Disconnected Operation in the Coda File System," ACM TOCS 10(1), 1992
ToolOpenAFS

Keeping caches valid

Caching is what makes a DFS fast. Every cache needs a rule for detecting staleness, and the three main approaches trade server state against network traffic.

Leases combine the other two. Like callbacks, they avoid constant polling. Like timeouts, they need no recovery: after a crash the server waits one lease period and every old promise has expired. Caching is not free, though. For write-heavy or widely shared files, invalidation traffic and stale reads can cost more than the cache saves.

PaperGray & Cheriton, "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency," SOSP 1989

GFS: chunks at scale

The Google File System (2003) started from different assumptions. Components fail constantly. Files are huge, often many gigabytes. Most writes append rather than overwrite. The workload is high-throughput batch reading, not low-latency interactive access. Those assumptions led to a very different design.

• Large chunks. Files are split into 64 MB chunks, each replicated (by default three times) on chunkservers that store them as ordinary Linux files.
• Single master. One master holds all metadata in memory: the namespace, the file-to-chunk map, and replica locations. It never handles file data.
• Separate control and data. Clients ask the master where chunks live, cache the answer, then exchange data directly with chunkservers.
• Leases for ordering. The master grants one replica a lease, making it primary. The primary chooses the order of mutations for that chunk, and the other replicas apply them in the same order.

A relaxed consistency model

GFS doesn't promise UNIX semantics. It defines what a file region looks like after each kind of mutation. A region is consistent if all replicas hold the same bytes, and defined if it is consistent and reflects one complete write.

Record append is the key operation: many clients append to one file at once, and GFS picks the offset, guaranteeing each record lands atomically at least once. Applications cope with duplicates and padding using checksums and record IDs. That suits producer–consumer pipelines and log merging, the workloads GFS was built for.

PapersGhemawat, Gobioff & Leung, "The Google File System," SOSP 2003 · McKusick & Quinlan, "GFS: Evolution on Fast-forward," CACM 53(3), 2010

Surviving failure

At thousands of machines, disks and servers fail every day. A DFS has to detect failures, keep enough copies, and repair itself without operators.

• Replication with re-replication. When a chunkserver stops sending heartbeats, the master schedules new copies of its chunks elsewhere, prioritizing chunks with the fewest surviving replicas.
• Placement across failure domains. Replicas go on different racks, so one switch or power failure can't take every copy.
• Checksums. Each chunkserver verifies data blocks on read, so silent disk corruption is caught and repaired from another replica.
• Metadata durability. The master logs every metadata change to an operation log replicated on other machines, with periodic checkpoints for fast restart. Read-only shadow masters can serve lookups while it recovers.
• Stale replica detection. A version number on each chunk lets the master spot replicas that missed updates while their server was down.

Replication vs. erasure coding

Three full copies are simple and fast to read but triple the storage. Erasure coding splits data into k fragments plus m parity fragments, and any k of the k + m fragments can rebuild the data. Large systems keep hot data replicated and convert colder data to erasure codes.

PapersFord et al., "Availability in Globally Distributed Storage Systems," OSDI 2010 · Huang et al., "Erasure Coding in Windows Azure Storage," USENIX ATC 2012 · Muralidhar et al., "f4: Facebook's Warm BLOB Storage System," OSDI 2014

Scaling metadata

A single master keeps the design simple, and GFS kept it out of the data path. But its memory caps the number of files, and every open still needs it. As clusters grew to billions of files, that became the limit. Later systems distribute metadata in three main ways:

• Federation: split the namespace among several independent metadata servers, each owning a subtree (HDFS federation).
• Metadata in a scalable database: store the namespace in a distributed key-value store or database (Google's Colossus on Bigtable, HopsFS on a NewSQL database, Meta's Tectonic on ZippyDB).
• Computed placement: compute where data lives from a hash function instead of looking it up. Ceph's CRUSH lets any client calculate object locations, with dynamic subtree partitioning for directories.

PapersWeil et al., "Ceph: A Scalable, High-Performance Distributed File System," OSDI 2006 · Weil et al., "CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data," SC 2006 · Niazi et al., "HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases," FAST 2017 · Pan et al., "Facebook's Tectonic Filesystem: Efficiency from Exascale," FAST 2021

Side by side

In practice

Tools and reading