data-systems-concepts

name

data-systems-concepts

description

Use when studying, explaining, designing, reviewing, or implementing data systems concepts including stream processing, replication, partitioning, transactions, isolation, consistency, consensus, failure detection, gossip, CRDTs, storage engines, pages, slotted pages, B-Trees, WAL, ARIES, MVCC, LSM trees, SSTables, compaction, Bloom filters, Merkle trees, checksums, compression, columnar storage, and database testing.

Data Systems Concepts

Goal: Help Codex turn data-systems questions into clear mental models, implementation exercises, and validation ideas.

Success means:

• The answer names the relevant system boundary, invariants, tradeoffs, and failure modes.
• The answer loads only the reference files needed for the current task.
• The answer points to credible online references when the user wants deeper study.

Stop when: The response gives a practical next step, implementation sketch, or review conclusion grounded in the relevant concepts.

Workflow

1. Identify the user's topic: dataflow, distributed coordination, storage engine internals, indexing, recovery, or testing.
2. Read references/00-topic-map.md when the best topic file is unclear.
3. Load one to three focused reference files for the task.
4. Return the smallest useful structure: mental model, key invariants, tradeoffs, implementation exercise, and tests or failure cases.
5. When writing code, choose a narrow artifact that demonstrates one concept and name the omissions explicitly.

Reference Groups

• High-level dataflow: topic map, architecture, logs, stream processing, watermarks, derived data.
• Distributed systems: replication, quorums, consistency, transactions, consensus, election, failure detection, partitioning, time, leases, snapshots, repair, CRDTs.
• Validation: data-system testing, model checking, deterministic simulation.
• Storage engines: pages, slotted pages, buffer pools, B-Trees, locking, secondary indexes, WAL, ARIES, MVCC, checkpoints, LSMs, SSTables, compaction, Bloom filters, Merkle trees, compression, checksums, columnar storage, blob separation.

Response Shape

Use this shape when it fits:

Mental model:
Key invariants:
Tradeoffs:
Implementation exercise:
Tests and failure cases:
References: