data-thinking

name	data-thinking
description	Data model selection, consistency trade-offs, data ownership, storage and processing decisions. Use when: choosing databases, designing data models, defining service data boundaries, replication/partitioning decisions, batch vs stream processing. Triggers: data model, database selection, consistency, replication, partitioning, CQRS, event sourcing, saga, aggregate design, transaction isolation

Data Thinking

Decision framework for data model, consistency, ownership, and processing choices. Not architecture structure (→ architect-thinking), not API surface design, not code-level design (→ design-philosophy).

Data Model Selection

Choose based on access pattern, not familiarity.

Model	When to use	Trade-offs
Relational	Many-to-many relationships, ad-hoc queries, strong consistency	Schema rigidity, impedance mismatch, join cost at scale
Document	Self-contained aggregates, variable schema, read-heavy denormalized	Poor cross-document joins, denormalization update anomalies
Graph	Highly connected data, relationship traversal, social/knowledge graphs	Query optimization complexity, fewer mature tools
Key-Value	Simple lookups, high throughput, no complex queries	No filtering by value, no relationships
Wide-column	Time-series, write-heavy, known query patterns	Must design around query patterns upfront, no ad-hoc queries

Decision driver: "What are the query patterns?" → model choice follows.

Consistency Spectrum

Not binary "strong vs eventual" — a spectrum with distinct cost profiles.

Level	Guarantee	Cost	Use when
Linearizability	Real-time total order	Highest latency, coordination required	Leader election, unique constraints
Sequential	Total order, not real-time	Lower than linearizable	Ordered operations without real-time need
Causal	Preserves cause-effect	Moderate, no global coordination	Most applications — good default
Eventual	Converges "eventually"	Lowest latency	High availability, tolerance for stale reads

Decision driver: "What are the consistency requirements per operation?" — not per system.

Replication Trade-offs

Strategy	Strength	Weakness
Single-leader	Simple, consistent reads from leader	Write bottleneck, failover complexity
Multi-leader	Write availability across regions	Conflict resolution complexity (LWW, merge, custom)
Leaderless (Dynamo-style)	High availability, no failover	Read-repair overhead, anti-entropy, quorum tuning

Partitioning Strategies

Strategy	Advantage	Risk
Key-range	Efficient range scans	Hot spots on sequential keys
Hash	Even distribution	No range scans, scatter-gather queries
Compound	First key hashed, second key range-ordered	Design complexity

Rebalancing: fixed partition count (pre-split) vs dynamic splitting. Avoid hash-mod-N (reshuffles everything on resize).

Transaction Isolation Levels

Level	Prevents	Allows
Read Committed	Dirty reads, dirty writes	Non-repeatable reads, write skew, phantoms
Snapshot Isolation (MVCC)	Non-repeatable reads	Write skew, phantoms
Serializable	All anomalies	Highest cost — actual serial, 2PL, or SSI

Know your database's actual implementation — naming varies (e.g., PostgreSQL "Repeatable Read" = Snapshot Isolation).

Batch vs Stream Processing

Dimension	Batch	Stream
Dataset	Bounded	Unbounded
Latency	Higher (minutes–hours)	Lower (seconds–ms)
Throughput	Higher	Varies
Error handling	Simpler (rerun job)	Complex (exactly-once, idempotency)
Examples	MapReduce, Spark	Kafka, Flink

• Lambda architecture: batch + stream layers → operational complexity of two systems
• Kappa architecture: stream only, replay from log → simpler but requires reprocessable log

Data Ownership & Service Boundaries

• Each service owns its data — shared database = hidden coupling
• Data disintegration drivers (split when): different change rates, different access patterns, different security requirements, independent scaling needs
• Data integration drivers (keep together when): transactional consistency required, data used together in queries, same ownership team

Decision driver: "Who owns this data?" → service boundary follows.

Distributed Data Patterns

Pattern	When	Trade-off
Saga (choreography)	Loosely coupled services, simple flows	No central visibility, difficult to debug
Saga (orchestration)	Complex flows, need central coordination	Single point of failure, orchestrator coupling
CQRS	Read and write patterns diverge significantly	Eventual consistency between models, sync complexity
Event Sourcing	Audit trail required, temporal queries, event-driven	Query complexity, storage cost, schema evolution of events

Aggregate Design (DDD)

• Consistency boundary = aggregate boundary — keep aggregates small
• Entity: identity matters, tracked across state changes
• Value Object: equality by attributes, immutable, replaceable
• Repository: one per aggregate root, abstracts data access
• Cross-aggregate references: by ID only, not object reference
• Cross-aggregate consistency: eventual (via domain events), not transactional

Schema Evolution

• Backward compatibility: new code reads old data
• Forward compatibility: old code reads new data
• Encoding formats: JSON (human-readable, no schema enforcement) → Avro/Protobuf (schema registry, compact, evolution rules) → use schema registry for Avro/Protobuf in production
• DB migrations: expand-then-contract (add new → migrate → remove old) to avoid downtime

Decision Checklist

Check	Question
Query patterns	What are the primary access patterns? → model choice
Consistency	What consistency level does each operation need? → isolation level
Ownership	Who owns this data? → service boundary
Change rate	How does this data change over time? → schema evolution strategy
Volume & velocity	Write/read ratio? Growth rate? → partitioning + replication
Processing model	Bounded or unbounded data? Latency requirement? → batch vs stream
Evolution	Can old code read new data and vice versa? → encoding format

Reference Books

Book	Author(s)	Key Contribution
Designing Data-Intensive Applications	Martin Kleppmann	Data models, replication, partitioning, consistency, batch/stream
Software Architecture: The Hard Parts	Neal Ford, Mark Richards et al.	Data ownership, decomposition drivers, distributed data patterns
Domain-Driven Design	Eric Evans	Aggregates, entities, value objects, repositories, bounded contexts