backend

# Backend Architecture ## Overview Backend architecture encompasses the server-side decisions that determine how a system stores data, exposes functionality, handles security, and scales under load. The choices made at this level -- API style, database type, caching strategy, authentication mechanism -- ripple through every layer of the application and are difficult to change once established. This skill provides a decision-making framework drawn from Martin Kleppmann's *Designing Data-Intensive Applications* and industry-proven patterns for building reliable, scalable, and maintainable backend systems. ## Knowledge Map ``` ┌─────────────────────────────────────────────────────────────────┐ │ API Layer │ │ REST, GraphQL, gRPC, WebSocket │ │ → How clients communicate with the backend │ ├─────────────────────────────────────────────────────────────────┤ │ Data Storage │ Caching │ │ Relational, Document, Graph, │ In-memory, Distributed, │ │ Key-Value, Time-Series │ CDN, HTTP caching │ │ → How data is persisted │ → How hot data is served │ ├─────────────────────────────────────────────────────────────────┤ │ Authentication & Authorization │ Background Processing │ │ OAuth 2.0, JWT, RBAC, ABAC, │ Job queues, schedulers, │ │ Multi-tenancy │ event-driven workers │ ├─────────────────────────────────────────────────────────────────┤ │ Rate Limiting & Throttling │ Observability │ │ Token bucket, sliding window, │ Logging, metrics, tracing, │ │ API quotas, circuit breakers │ health checks, alerting │ └─────────────────────────────────────────────────────────────────┘ ``` ## Choosing an API Style | Criterion | REST | GraphQL | gRPC | WebSocket | |-----------|------|---------|------|-----------| | **Best for** | CRUD resources, public APIs | Flexible queries, mobile clients | Internal microservices, high throughput | Real-time bidirectional communication | | **Data format** | JSON (typically) | JSON | Protobuf (binary) | Any (JSON, binary) | | **Contract** | OpenAPI / Swagger | Schema (SDL) | .proto files | No standard schema | | **Caching** | HTTP caching (excellent) | Harder (POST-based) | No HTTP caching | Not cacheable | | **Streaming** | SSE (server-only) | Subscriptions | Bidirectional streaming | Full-duplex native | | **Browser support** | Native | Native | Requires gRPC-Web proxy | Native | | **Learning curve** | Low | Medium | Medium-High | Low-Medium | | **Over/under-fetching** | Common problem | Solved by design | Defined per RPC | N/A | | **Tooling maturity** | Excellent | Good | Good (growing) | Moderate | **Decision heuristic:** - Default to **REST** for public APIs and simple CRUD services. - Choose **GraphQL** when clients need flexible, aggregated queries across multiple resources (especially mobile). - Choose **gRPC** for internal service-to-service communication where latency and throughput matter. - Choose **WebSocket** when you need real-time, bidirectional data flow (chat, live dashboards, collaborative editing). - Many systems combine styles: REST for public API, gRPC internally, WebSocket for real-time features. ## Choosing a Database Type | Criterion | Relational (SQL) | Document (NoSQL) | Graph | Key-Value | Time-Series | |-----------|------------------|-------------------|-------|-----------|-------------| | **Best for** | Structured data, complex joins, ACID transactions | Flexible schemas, nested data, rapid iteration | Highly connected data, relationship traversal | Simple lookups, caching, session storage | Metrics, IoT, logs, financial ticks | | **Examples** | PostgreSQL, MySQL, SQL Server | MongoDB, CouchDB, DynamoDB | Neo4j, Amazon Neptune, ArangoDB | Redis, Memcached, DynamoDB | InfluxDB, TimescaleDB, Prometheus | | **Schema** | Strict (schema-on-write) | Flexible (schema-on-read) | Property graph / RDF | Schema-free | Tag + field model | | **Scaling** | Vertical (horizontal with sharding) | Horizontal (built-in) | Vertical (some horizontal) | Horizontal (built-in) | Horizontal (built-in) | | **Transactions** | Full ACID | Limited (document-level) | Varies by product | None (typically) | None (typically) | | **Query language** | SQL | Vendor-specific (MQL, etc.) | Cypher, Gremlin, SPARQL | GET/SET commands | InfluxQL, Flux, SQL | | **Joins** | Excellent | Poor (application-level) | Excellent (traversals) | None | Limited | **Decision heuristic:** - Default to **relational** (PostgreSQL) when data is structured and relationships matter. - Choose **document** when schema flexibility and developer velocity are priorities, and joins are rare. - Choose **graph** when the primary queries traverse relationships (social networks, recommendations, fraud detection). - Choose **key-value** for caching, sessions, and simple lookup-by-key workloads. - Choose **time-series** for append-heavy, time-stamped data with downsampling and retention needs. - Polyglot persistence is common: use the right database for each bounded context. ## Backend Architecture Concerns ### Rate Limiting & Throttling Protect backend services from abuse and overload: - **Token Bucket** -- allows bursts up to a configured capacity, refills at a steady rate. - **Sliding Window** -- counts requests in a rolling time window for smoother limiting. - **Fixed Window** -- simple counter per time window (risk of burst at window boundaries). - **Leaky Bucket** -- processes requests at a constant rate, queuing excess. - Implement at the API gateway layer for consistency across services. ### Background Processing Offload long-running or non-urgent work from the request/response cycle: - **Job queues** (Sidekiq, Celery, Hangfire, BullMQ) -- enqueue work, process asynchronously. - **Scheduled jobs** (cron, Quartz, Hangfire recurring) -- time-triggered processing. - **Event-driven workers** -- react to domain events from a message broker. - **Batch processing** -- periodic bulk operations (ETL, report generation). - Always design for idempotency -- workers may process the same job more than once. ### Observability The three pillars of observability, plus health monitoring: - **Logging** -- structured logs (JSON) with correlation IDs for request tracing. - **Metrics** -- counters, gauges, histograms (request rate, error rate, latency percentiles). - **Distributed Tracing** -- end-to-end trace across services (OpenTelemetry, Jaeger, Zipkin). - **Health checks** -- liveness (is the process running?) and readiness (can it serve traffic?). - **Alerting** -- thresholds on key metrics (error rate > 1%, p99 latency > 500ms). ## Canonical Reference - *Designing Data-Intensive Applications* by Martin Kleppmann -- the definitive guide to data storage, replication, partitioning, encoding, and distributed system trade-offs. Essential reading for any backend architect. ## Sub-Skills - `dev/backend/data-modeling` -- Data modeling and database architecture patterns - `dev/backend/api-design` -- API design patterns for REST, GraphQL, gRPC, WebSocket - `dev/backend/caching` -- Caching strategies and patterns - `dev/backend/authentication` -- Authentication and authorization patterns ## Best Practices - Start with a monolith and extract services only when complexity demands it -- premature microservices add coordination cost without proportional benefit. - Choose boring technology by default. PostgreSQL, Redis, and a well-designed REST API solve the vast majority of backend problems. - Design for failure: every network call can fail, every database can be slow. Use timeouts, retries with backoff, circuit breakers, and fallbacks. - Make operations idempotent wherever possible -- especially for writes, background jobs, and event handlers. - Instrument everything from day one. Adding observability retroactively is far more expensive than building it in. - Treat API contracts as public commitments: version explicitly, deprecate gracefully, never break existing clients without a migration path.