| name | distributed-storage |
| description | Distributed storage systems design and operation for cloud platforms. Covers the GFS/HDFS block-and-master pattern, object storage (Swift/S3) with consistent hashing and eventual consistency, block storage semantics, replication vs erasure coding, the CAP theorem in practice, read-repair and anti-entropy, snapshot chains, and the GFS/BigTable/Spanner evolution. Use when designing a storage subsystem, choosing between object/block/file, or reviewing a replication and consistency strategy. |
| type | skill |
| category | cloud-systems |
| status | stable |
| origin | tibsfox |
| modified | false |
| first_seen | "2026-04-12T00:00:00.000Z" |
| first_path | examples/skills/cloud-systems/distributed-storage/SKILL.md |
| superseded_by | null |
Distributed Storage
Distributed storage is where durability, consistency, latency, and cost collide. Every design choice trades among them, and the trade-offs cannot be hidden from applications for long. This skill covers the landmark systems and the recurring patterns they embody, from the GFS master-chunkserver split that defined cloud-scale storage, to the Dynamo-style consistent-hash rings that power modern object stores, to the Spanner-style externally consistent databases built on TrueTime.
Agent affinity: ghemawat (GFS and storage systems craftsmanship), dean (BigTable, Spanner, and the evolution from GFS), decandia (Dynamo and eventually consistent stores)
Concept IDs: cloud-cinder-block-storage, cloud-swift-glance-object-image, cloud-nova-instances
The Three Storage Shapes
Cloud platforms expose storage in three shapes, each with a different access model and consistency profile.
Object storage. Immutable blobs with rich metadata, accessed by key, usually over HTTP. Examples: Amazon S3, OpenStack Swift, Google Cloud Storage. Consistent hashing distributes objects across nodes; replication or erasure coding provides durability. Objects are typically append-or-replace — partial updates are not the native operation. Best for: media, backups, logs, analytics input, static web content.
Block storage. Virtual disks presented as block devices to virtual machines. Examples: Amazon EBS, OpenStack Cinder, Google Persistent Disk. Provides the POSIX-ish semantics of a local disk but with the durability and mobility of a network service. Typically attached to a single instance at a time. Best for: databases, filesystems, anything that expects random-access block semantics.
File storage. Shared filesystem accessed via NFS, SMB, or similar protocols. Examples: Amazon EFS, OpenStack Manila, GCS Filestore. Multiple clients mount the same filesystem and see each other's writes. Best for: legacy applications expecting a POSIX filesystem and shared access.
These are not interchangeable. Picking the wrong shape — object where you need block, block where you need file, file where you need object — produces workarounds that dominate the cost of running the system.
The GFS Pattern: Master and Chunkservers
The Google File System (Ghemawat, Gobioff, Leung, 2003) introduced the architecture that most large distributed filesystems still use:
- Single master. Holds all metadata: namespace, access control, mapping from files to chunks, mapping from chunks to chunkservers. Metadata fits in RAM for the scales GFS was built for.
- Chunkservers. Hold actual file data in 64 MB chunks (later 64-256 MB in successors). Each chunk is replicated 3 times across different failure domains.
- Query the master for chunk locations, then read and write directly from/to chunkservers. The master is not on the data path.
