system-design

name	system-design
description	Use when designing a new service from scratch, writing a tech spec or RFC, selecting a database or communication pattern, estimating capacity, or reviewing a design for scalability and reliability gaps.

System Design

A structured end-to-end guide for designing systems from scratch, making architecture decisions, estimating capacity, and building for scalability and reliability.

When to Activate

• Designing a new service, platform, or system from scratch
• Writing a design document, tech spec, or RFC
• Making a technology selection or architecture decision
• Writing an Architecture Decision Record (ADR)
• Reviewing a design for scalability, reliability, or security gaps
• Planning capacity for a feature expected to handle significant load

---

Design Process

Follow these six steps in order. Each step informs the next — skipping steps leads to rework.

1. Clarify requirements — Gather functional requirements (what it does), non-functional requirements (scale, latency, availability, durability), and constraints (budget, existing stack, team size, timeline).
2. Estimate scale — QPS, storage, bandwidth. See Capacity Estimation section.
3. Define API contracts — What APIs does the system expose? What are the request/response shapes, authentication mechanisms, and versioning strategy? Reference the api-design skill for REST/gRPC conventions.
4. Design the data model — Define entities, relationships, access patterns, and storage technology. Choose SQL vs NoSQL based on the Technology Selection Matrix.
5. Design components — Produce a High-Level Design (HLD) diagram, assign service responsibilities, and define communication patterns (sync vs async).
6. Identify bottlenecks — Find single points of failure, scaling limits, hot partitions, and cascading failure risks before the design is locked.

Quick Decision Framework

Non-functional requirement	Design implication
High read throughput	Read replicas, caching layer
High write throughput	Sharding, async writes, CQRS
Low latency	CDN, in-process cache, co-location
High availability	Multi-AZ, load balancing, circuit breakers
Strong consistency	Single-leader DB, distributed transactions (use carefully)
Eventual consistency	Event-driven, CQRS + event sourcing acceptable

---

Capacity Estimation

Back-of-envelope estimation before any design work. Use these numbers to size components and catch obvious scaling problems early.

Back-of-Envelope Template

# Traffic
Daily Active Users (DAU): X
Requests per user per day: Y
QPS (avg) = X * Y / 86400
QPS (peak) = avg * 3x

# Storage
Data per request: Z bytes
Daily new data = QPS_avg * 86400 * Z
1 year storage = daily * 365
With replication (3x): total * 3

# Bandwidth
Inbound  = QPS * avg_request_size
Outbound = QPS * avg_response_size

Example Estimation

# Twitter-scale write path example
DAU = 100M
Tweets per user per day = 0.5
QPS (avg) = 100M * 0.5 / 86400 ≈ 580 QPS
QPS (peak) = 580 * 3 ≈ 1,750 QPS

Tweet size = 300 bytes
Daily new data = 580 * 86400 * 300 ≈ 15 GB/day
1 year = 15 * 365 ≈ 5.5 TB
With 3x replication ≈ 16.5 TB/year

Latency Reference Numbers

Operation	Latency
L1 cache	~1 ns
L2 cache	~10 ns
RAM read	~100 ns
SSD read	~100 µs
Network round trip (same DC)	~500 µs
Network round trip (cross-region)	~30–100 ms
HDD seek	~10 ms

Internalize these numbers. When someone says "just add a DB call", that is ~100 µs on SSD minimum — more if the query is complex or the DB is cross-region.

---

High-Level Design

Component Types

Component	Responsibility	When to add
Load Balancer	Distribute traffic, health checks, SSL termination	Multiple app instances
API Gateway	Auth, rate limiting, routing, protocol translation	Public-facing APIs, microservices
Application Server	Business logic	Always
Cache (Redis/Memcached)	Reduce DB reads, session storage	Hot data, session state
Relational DB	ACID transactions, structured data	Most workloads
NoSQL DB	Flexible schema, high write throughput, time series	Specific access patterns
Message Queue	Async processing, decoupling, fan-out	Background jobs, event-driven flows
CDN	Static asset delivery, edge caching	Web apps, high-read global content
Object Storage (S3)	Files, images, backups	Binary data, large files
Search Engine (Elasticsearch)	Full-text search, complex queries	Search, log analytics

Diagram Conventions (text-based)

Use ASCII block diagrams when Mermaid is unavailable. Vertical lines show primary request paths; horizontal lines show async or secondary flows.

Client
  │
  ▼
[CDN]──────────────────────────────────┐
  │                                    │
  ▼                                  Static
[Load Balancer]                       Assets
  │
  ├──► [App Server 1]
  ├──► [App Server 2]   ──► [Cache (Redis)]
  └──► [App Server N]
              │
              ▼
       [Primary DB] ──► [Read Replica 1]
                    ──► [Read Replica 2]

Adding a Message Queue

[App Server]
     │
     ▼
[Message Queue (Kafka/SQS)]
     │
     ├──► [Worker 1: Email notifications]
     ├──► [Worker 2: Analytics pipeline]
     └──► [Worker 3: Search indexing]

Decouple producers and consumers. App servers enqueue and return immediately; workers process asynchronously without blocking the request path.

---

Low-Level Design

Sequence Diagrams (Mermaid)

Use Mermaid sequence diagrams to describe multi-service interactions. Always show the happy path first, then error cases separately.

Example: Authentication flow

sequenceDiagram
    participant Client
    participant API
    participant AuthService
    participant DB

    Client->>API: POST /auth/login {email, password}
    API->>AuthService: validate(email, password)
    AuthService->>DB: SELECT user WHERE email=?
    DB-->>AuthService: user record
    AuthService->>AuthService: bcrypt.verify(password, hash)
    AuthService-->>API: {userId, roles}
    API-->>Client: {access_token, refresh_token}

Example: Token refresh error path

sequenceDiagram
    participant Client
    participant API
    participant AuthService

    Client->>API: POST /auth/refresh {refresh_token}
    API->>AuthService: validate_refresh(token)
    AuthService-->>API: TokenExpiredError
    API-->>Client: 401 Unauthorized {error: "token_expired"}

Class/Interface Design

Define interfaces at service boundaries, not at implementation boundaries. Depend on abstractions; never depend on concrete classes across module boundaries.

UserRepository interface

from abc import ABC, abstractmethod
from typing import Optional
from uuid import UUID

class UserRepository(ABC):

    @abstractmethod
    def find_by_id(self, user_id: UUID) -> Optional["User"]:
        ...

    @abstractmethod
    def find_by_email(self, email: str) -> Optional["User"]:
        ...

    @abstractmethod
    def save(self, user: "User") -> "User":
        ...

    @abstractmethod
    def delete(self, user_id: UUID) -> None:
        ...

PostgresUserRepository implementation

class PostgresUserRepository(UserRepository):

    def __init__(self, db: Session):
        self._db = db

    def find_by_id(self, user_id: UUID) -> Optional[User]:
        return self._db.query(UserModel).filter_by(id=user_id).first()

    def find_by_email(self, email: str) -> Optional[User]:
        return self._db.query(UserModel).filter_by(email=email).first()

    def save(self, user: User) -> User:
        self._db.merge(user)
        self._db.commit()
        return user

    def delete(self, user_id: UUID) -> None:
        self._db.query(UserModel).filter_by(id=user_id).delete()
        self._db.commit()

The service layer only imports UserRepository. Swapping Postgres for DynamoDB requires only a new implementation class — no changes to the service.

State Machines

Model entities as state machines when they have a well-defined lifecycle with discrete states and transitions. Common examples: orders, subscriptions, payments, onboarding flows.

When to use a state machine:

• The entity has more than two states
• Transitions have side effects (send email, charge card, create record)
• Invalid transitions must be rejected

Order lifecycle example

Current State	Event	Next State	Action
PENDING	payment_confirmed	PAID	Send order confirmation email
PAID	items_shipped	SHIPPED	Send shipping notification
SHIPPED	delivery_confirmed	DELIVERED	Release funds to merchant
PAID	cancellation_requested	CANCELLED	Issue refund
SHIPPED	cancellation_requested	REFUND_PENDING	Initiate return process
DELIVERED	refund_requested	REFUND_PENDING	Start refund review

Store the current state in the database. Reject any event that does not have a valid transition from the current state. Log every transition with timestamp and actor.

---

Architecture Decision Records (ADRs)

An ADR captures the context, decision, and consequences of a significant architecture choice. Write one whenever you make a decision that would be expensive or disruptive to reverse.

Template

# ADR-NNNN: [Short title]

## Status
[Proposed | Accepted | Deprecated | Superseded by ADR-XXXX]

## Context
[What is the problem? What forces are at play?]

## Decision
[What have we decided to do?]

## Consequences
### Positive
- ...

### Negative
- ...

## Alternatives Considered
| Option | Pros | Cons | Reason rejected |
|--------|------|------|-----------------|
| ...    | ...  | ...  | ...             |

ADR Conventions

• File naming: docs/adr/0001-use-postgres-for-primary-store.md
• Numbering: Sequential, zero-padded to four digits. Never renumber existing ADRs.
• Lifecycle: Proposed → Accepted → (Deprecated | Superseded)
• Superseded ADRs: Keep the file. Add a note at the top: "Superseded by ADR-0012." Link forward, never delete.
• Write an ADR when: changing a primary database, switching communication patterns, adopting a new framework, changing authentication strategy, or adding a new external dependency that will be hard to remove.
• Do not write an ADR for: library version bumps, minor refactors, tooling preferences with no architectural impact.

Example ADR

# ADR-0003: Use PostgreSQL for primary data store

## Status
Accepted

## Context
We need a primary relational store. The team has strong SQL expertise.
Our access patterns are mostly relational with complex join queries.
We need ACID transactions for financial operations.

## Decision
Use PostgreSQL 15 as the primary relational database.

## Consequences
### Positive
- Full ACID compliance
- Rich query planner and index types (BRIN, GIN, partial)
- Strong community and tooling ecosystem

### Negative
- Vertical scaling only for writes (mitigated with read replicas)
- Schema migrations require care at scale

## Alternatives Considered
| Option      | Pros                        | Cons                             | Reason rejected               |
|-------------|-----------------------------|----------------------------------|-------------------------------|
| MySQL 8     | Widely supported            | Weaker JSON support, less ANSI   | Team unfamiliar, fewer features |
| MongoDB     | Flexible schema             | No multi-doc ACID, weak joins    | Access patterns are relational |
| CockroachDB | Distributed SQL, geo-local  | Operational complexity, cost     | Premature for current scale   |

---

Technology Selection Matrix

SQL vs NoSQL

Criterion	SQL (PostgreSQL)	Document (MongoDB)	Key-Value (Redis)	Column (Cassandra)
ACID transactions	Full	Limited	No	Lightweight
Query flexibility	High (joins, aggregates)	Medium	Low	Low
Schema	Strict	Flexible	None	Flexible
Scale-out	Vertical + read replicas	Horizontal	Horizontal	Horizontal
Best for	Most apps, financial data	Flexible documents	Cache, sessions	High-write, time series

Default: Start with PostgreSQL. Move to NoSQL only when you have a concrete access pattern that PostgreSQL cannot handle at scale.

Sync vs Async Communication

Pattern	Latency	Coupling	Best for
REST/gRPC (sync)	Low	Tight	Request/response, queries
Message queue (async)	Higher	Loose	Background jobs, fan-out, retry
Event streaming (Kafka)	Medium	Very loose	Audit log, real-time analytics, event sourcing

Rule of thumb: If the caller needs the result to continue, use sync. If the caller only needs to know the work was accepted, use async.

Monolith vs Microservices

Factor	Monolith	Microservices
Team size	< 10 engineers	> 10 engineers with clear domain ownership
Deploy complexity	Low	High (orchestration required)
Data isolation	Shared DB (simple)	DB per service (complex)
Scalability	Scale whole app	Scale individual services
Start with	Always	Only when monolith has real pain points

Default: Start with a modular monolith. Extract services only when a specific bounded context has meaningfully different scaling or deployment needs.

---

Scalability Patterns

Horizontal Scaling

Make services stateless so any instance can handle any request.

• Store session state in Redis, not in-process memory
• Store uploaded files in object storage (S3), not on the local filesystem
• Store configuration in environment variables or a config service
• Never use sticky sessions in a load balancer unless absolutely required

# Stateless service checklist
- No in-process session state
- No local file system dependencies
- Idempotent request handling (safe to retry)
- Config from environment, not hardcoded

Read Replicas

Route read-heavy queries to replicas to offload the primary.

[App Server]
  │
  ├──[WRITE]──► [Primary DB]
  │
  └──[READ]───► [Read Replica 1]
               [Read Replica 2]

• Accept replication lag: reads from replicas may be slightly stale
• Use the primary for reads immediately following a write (read-your-writes consistency)
• Monitor replication lag — alert if it exceeds your SLA tolerance

Sharding

Partition data across multiple database nodes when a single node cannot handle write throughput or storage.

• Hash sharding: Apply a hash function to the shard key (e.g., user_id % N). Provides even distribution. Use consistent hashing to reduce rebalancing cost.
• Range sharding: Partition by ordered key (e.g., created_at by month). Efficient for time-series scans. Risk: hot partitions at the current time range.
• Directory sharding: A lookup table maps keys to shards. Flexible but lookup table becomes a bottleneck.

Sharding problems to plan for:

• Cross-shard queries require scatter-gather — expensive
• Rebalancing when adding shards is operationally complex
• Unique ID generation must be shard-aware (use UUIDs or Snowflake IDs)

CQRS (Command Query Responsibility Segregation)

Separate the write model (commands) from the read model (queries).

                    ┌────────────────────────┐
Write Path          │                        │   Read Path
                    │                        │
[Command] ──► [Write Model] ──► [Event Bus] ──► [Projections] ──► [Read Model]
             (normalized,                        (denormalized,
              ACID DB)                           query-optimized)

• Write model: Normalized, strongly consistent, ACID transactions
• Read model: Denormalized projections tailored to specific query shapes
• Event sourcing on the write side: Store events (facts) rather than current state; derive state by replaying events
• Use when: Read and write access patterns are fundamentally different, or you need audit history

Caching Layers

Layer	Technology	Scope	Invalidation
L1	In-process LRU (e.g., functools.lru_cache)	Single process	TTL or restart
L2	Distributed cache (Redis, Memcached)	All app instances	TTL, event-driven, write-through
L3	CDN (Cloudflare, CloudFront)	Public content, global edge	Cache-Control headers, purge API

Cache invalidation strategies:

• TTL (time-to-live): Simple, tolerates stale data. Good for reference data.
• Write-through: Write to cache and DB simultaneously. Cache is never stale but adds write latency.
• Event-driven invalidation: On write, publish an event; consumers invalidate their cache entries. Complex but accurate.
• Cache-aside (lazy loading): Read from cache; on miss, read from DB and populate cache. Most common pattern.

---

Reliability Patterns

Circuit Breaker

Prevents cascading failures when a downstream dependency degrades or fails.

States:

• Closed (normal): Requests flow through. Failures are counted.
• Open (failing): Circuit trips after threshold failures. Requests are rejected immediately (fast fail) without calling the downstream.
• Half-Open (probing): After a cooldown period, a small number of requests are allowed through. If they succeed, the circuit closes. If they fail, it reopens.

[App] ──► [Circuit Breaker] ──► [Downstream Service]
                │
                └── If OPEN: return fallback immediately

Libraries:

• Python: circuitbreaker
• Node.js: opossum
• Go: gobreaker
• Java: Resilience4j

Retry with Exponential Backoff

wait = base_delay * (2 ^ attempt) + jitter
max_attempts = 3–5

• Jitter: Add random noise (±50% of computed wait) to prevent the thundering herd problem — all clients retrying simultaneously
• Only retry idempotent operations: GET, PUT, DELETE are safe. POST may not be unless you add idempotency keys.
• Set a budget: Total retry time must be less than the upstream timeout.

import random
import time

def retry_with_backoff(fn, max_attempts=4, base_delay=0.5):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RetryableError as e:
            if attempt == max_attempts - 1:
                raise
            wait = base_delay * (2 ** attempt) + random.uniform(0, base_delay)
            time.sleep(wait)

Bulkhead

Isolate resource pools so that saturation in one consumer does not exhaust shared resources.

[App Server]
  │
  ├── Thread Pool A (50 threads) ──► [Payment Service]
  ├── Thread Pool B (20 threads) ──► [Inventory Service]
  └── Thread Pool C (10 threads) ──► [Recommendation Service]

• If the Recommendation Service hangs, Thread Pool C exhausts, but Thread Pool A and B are unaffected
• Apply bulkheads for any downstream dependency that could be slow or unreliable
• Size each pool based on expected concurrency and dependency SLA

Graceful Degradation

Return useful (possibly stale) responses when live data is unavailable.

Scenario	Degraded response
Inventory service down	Show product listings from cache, hide real-time stock count
Recommendation engine down	Show static "popular items" list
Search service down	Disable search box, show browse-by-category fallback
Payment service degraded	Queue payment, confirm async, inform user

Implementation pattern: Wrap dependency calls in a try/except or circuit breaker. On failure, return the last cached value or a safe default.

Timeout Hierarchy

Always set explicit timeouts at every layer. Never rely on defaults — most frameworks default to infinite or very large timeouts.

Layer	Recommended timeout
User-facing API (client → server)	500 ms – 2 s
Internal service-to-service	100 – 500 ms
Database queries	5 – 30 s (enforce with statement_timeout)
Async job processing	Set per-job based on SLA

Timeout budget rule: The sum of downstream timeouts in a synchronous call chain must be less than the upstream timeout. If service A calls B calls C, then timeout(C) + timeout(B overhead) < timeout(B), and so on up the chain.

---

Red Flags

• Starting with the data model instead of API contracts — the schema should follow the access patterns, not the other way around; design API and user journeys first, then derive the schema
• Microservices for a greenfield system — premature decomposition introduces distributed-systems overhead before domain boundaries are understood; start as a modular monolith
• Ignoring the CAP theorem tradeoff — every distributed system is either CP or AP under partition; make the choice explicit, document it, and design client error handling accordingly
• Sharding without modeling both write and read access patterns — a shard key that distributes writes evenly can cause hot-spot reads; model all query patterns before committing to a key
• Read replicas for all reads — replication lag means replica reads return stale data; never read from a replica immediately after a write in the same request flow
• No failure-mode analysis at design time — designing for the happy path and deferring failure handling to implementation produces brittle systems; define timeout, retry, and circuit-breaker policies in the design
• ADR skipped because the decision "feels obvious" — obvious decisions are often the hardest to reverse; write the ADR before implementation starts, including rejected alternatives

Checklist

• Requirements clarified: functional, non-functional, constraints
• Capacity estimated: QPS, storage, bandwidth for 1x and 10x load
• Single points of failure identified and mitigated
• Database choice justified with a decision table or ADR
• Caching strategy defined for read-heavy paths
• Async communication used for non-blocking operations
• Authentication and authorization designed (not an afterthought)
• ADR written for every significant technology or architecture decision
• Monitoring and alerting considered in the design
• Disaster recovery: RTO and RPO defined
• Timeout, retry, and circuit breaker policies defined for all external calls
• Design reviewed by at least one other engineer