system-design

name	system-design
description	Use when the user asks to "design a system", "evaluate architecture", "scale a service", "choose between databases", "design for high availability", "review a system design", or discusses distributed systems, microservices, data modeling, or architectural trade-offs. Provides senior engineering guidance for system architecture decisions.

System Design

Architecture is the set of decisions that are hardest to reverse. Every system design session is an exercise in identifying which decisions matter most, making them explicitly with known trade-offs, and deferring everything else.

When to Activate

• Starting a new service or major feature
• Evaluating scaling options for an existing system
• Reviewing an architecture proposal
• Choosing between data storage solutions
• Designing for high availability or fault tolerance
• Decomposing a monolith

Core Principles

Start with Requirements, Not Solutions

Before drawing boxes and arrows, establish constraints:

• Scale: peak RPS, p99 latency target, data volume
• Availability: acceptable downtime (99.9% = 8.7 hrs/year, 99.99% = 52 min/year)
• Consistency: is stale data acceptable? For how long? For which operations?
• Operational complexity: what can the team actually run?

Requirements constrain the solution space. A design optimized for 1,000 RPS is wrong for 1M RPS, and vice versa. Get numbers before choosing tools.

CAP and Consistency Models

Under network partition (which always happens eventually), a distributed system must choose between Consistency and Availability. This is not a one-time system-wide choice — it is made per-operation.

Common consistency models ranked weakest to strongest:

1. Eventual consistency — all replicas converge, given enough time. High availability, stale reads possible.
2. Read-your-writes — a client always sees its own updates. Common minimum requirement for UX.
3. Monotonic reads — once a value is read, subsequent reads return the same or newer values.
4. Causal consistency — causally related writes appear in order to all readers.
5. Linearizability — reads always return the most recently written value. Highest cost, highest safety.

Choose the weakest model that satisfies your correctness requirements. Linearizability requires coordination that limits throughput.

The Four Scaling Approaches

1. Vertical scaling (bigger machine) — simplest, has a ceiling, no code changes
2. Horizontal scaling (more machines) — requires stateless services or distributed state
3. Read replicas — offload reads to replicas; effective when read:write ratio is high
4. Sharding/partitioning — distribute writes; adds significant operational complexity

Apply in order. Most systems never need sharding. Most systems that need sharding need it on one hot table.

Data Storage Selection

Requirement	Candidate
Relational data with transactions	PostgreSQL
High-write time series	InfluxDB, TimescaleDB
Document-oriented, flexible schema	MongoDB (with caution)
Low-latency key-value	Redis, DynamoDB
Full-text search	Elasticsearch, Typesense
Graph relationships	Neo4j
Object/blob storage	S3, GCS
Message queue	Kafka (high-throughput), SQS (managed simplicity)

Default to PostgreSQL and a managed object store. Move off this baseline when a specific bottleneck is proven, not anticipated.

Distributed Systems Patterns

Synchronous vs Asynchronous Communication

Synchronous (HTTP/gRPC): simple, immediate feedback, couples availability Asynchronous (queues/events): decouples availability, harder to debug, eventual consistency

Use sync for: user-facing reads, simple request/response Use async for: long-running work, fan-out to multiple consumers, cross-service operations that should not block

Common Failure Patterns

Cascade failure: service A calls B, B calls C; C slows → B threads exhaust → A threads exhaust → full outage. Mitigation: circuit breakers, bulkheads, timeouts on every outbound call.

Thundering herd: cache expires, 10,000 requests hit the database simultaneously. Mitigation: cache stampede protection (probabilistic early expiration, locks).

Split-brain: two nodes each believe they are the primary. Mitigation: consensus algorithms (Raft, Paxos), odd node counts, fencing tokens.

Hot partition: one shard receives all traffic because the partition key is poor (e.g., user_id when one user has 1M events). Mitigation: write sharding (prefix hash), read fan-out.

Idempotency

Every operation that can be retried must be idempotent. Assign idempotency keys at the edge (client-generated UUID). Store keys with a TTL. On duplicate request, return the stored result without re-executing.

Non-idempotent operations (charge a card, send an email) become idempotent through this pattern. Do not skip it.

High Availability Design

The Three Axes

1. Redundancy: eliminate single points of failure — N+1 for every stateful component
2. Graceful degradation: define which features can be disabled under load (circuit breakers to stubs)
3. Observability: you cannot restore what you cannot measure — metrics, logs, traces before launch

Health Checks

Three types, with different purposes:

• Liveness: is the process running? (if not, restart it)
• Readiness: is the instance ready to serve traffic? (if not, remove from load balancer pool)
• Deep health: are downstream dependencies healthy? (use only for monitoring, never for liveness — cascades kill)

Practical Guidance

Monolith vs Microservices

Start monolithic. Extract services when you have:

• Independent scaling requirements for a specific component
• Team boundaries that map to service boundaries
• Deployment independence that justifies operational overhead

Microservices are an organizational pattern as much as a technical one. The overhead (distributed tracing, service discovery, network latency, failure isolation) is real. Don't pay it until the benefit is proven.

Back-of-the-Envelope Estimates

Operation	Approximate latency
L1 cache read	0.5 ns
L2 cache read	7 ns
RAM read	100 ns
SSD random read	150 µs
Network roundtrip (same DC)	0.5 ms
Network roundtrip (cross-region)	150 ms
HDD seek	10 ms

Memorize these. A design that requires 10 sequential cross-region calls per request will never hit 200ms p99.

Gotchas

1. Premature optimization of the wrong layer: engineers optimize the application tier while the query has a missing index. Profile the actual bottleneck — databases are almost always the constraint.

2. Distributed systems are not reliable by default: adding more services adds more failure modes, not more reliability. Reliability comes from design, not topology.

3. Eventual consistency surprises users: "your balance is temporarily unavailable" is not acceptable for a payment system. Map consistency requirements per operation, not per system.

4. Schema migrations are the hardest part of scaling: a 10ms migration on a 10GB table is a 10-hour migration on a 10TB table. Design for online migrations (expand/contract pattern) from the start.

5. Load testing with production traffic patterns matters: a benchmark with uniform request distribution misses the hot paths that cause real outages. Replay production traces.

6. N+1 queries at scale are catastrophic: an ORM that issues N queries for N records looks fine at 100 records and destroys a database at 100,000. Instrument query counts, not just latency.

7. Caching without cache invalidation is a liability: stale cache + long TTL = incorrect data served for hours. Design invalidation before implementing the cache.

Integration

• api-design — the interface contract for services you design here
• performance-engineering — how to measure whether the design performs as expected
• security-engineering — threat model the system you design
• incident-response — design for observability and failure recovery

References

• Architectural Patterns Reference — read when choosing between specific architectural patterns (CQRS, event sourcing, saga, outbox)
• Design Validator Script — checklist validator for system design reviews

Skill Metadata

Created: 2026-04-10 Version: 1.0.0