latency-engineering

# AI Skill: Latency Engineering This skill guides the artificial intelligence to analyze, model, and reduce end-to-end latency in software systems, from theoretical foundations (queuing theory) to low-level implementation techniques, based on the work *Latency: Reduce delay in software systems* (Pekka Enberg, Manning). --- ## 📏 1. Fundamental Laws of Latency ### Little's Law Connects latency (L), throughput (X), and concurrency (N): **N = L × X** - Use it to size connection pools, thread pools, and buffers: at 1,000 req/s and a target latency of 50 ms, the system needs roughly 50 concurrent requests. - If throughput saturates and N grows without bound, all extra capacity goes into the queue (latency explodes). - Mind the assumptions: it holds only in steady state; unstable systems violate the theorem's premise. ### Amdahl's Law Speedup limit from parallelization: **Speedup = 1 / (F + (1 − F)/N)**, where F is the serial fraction. - The serial fraction is the watershed: even with infinite cores, maximum speedup is 1/F. - Use it to estimate diminishing returns from parallelization before investing in more hardware. ### Queuing Theory & Utilization - Latency grows **non-linearly** as utilization (ρ) approaches 1 (M/M/1 queue: latency ∝ 1/(1−ρ)). - Rule of thumb: keep latency-critical services below roughly 70–80% utilization; above that, p99 grows exponentially. --- ## 📊 2. Tail Latency and Coordinated Omission - **Percentiles, not averages**: Measure and optimize p99/p99.9, because at scale a single user makes few requests and will be affected by the tail. - **Distributed systems amplify the tail**: a pipeline of 10 services with a 100 ms p99 can have a far worse end-to-end tail latency (fan-out multiplies the chance of a slow path). - **Coordinated Omission (Gil Tene)**: load generators that synchronize requests (closed-loop) hide stall periods and drastically underestimate the tail. Fix it by reporting the latencies of requests that *should* have been sent during the stall. - **Head-of-line blocking**: one slow request in the queue (even single-threaded, as in Node.js, or in common framework filters) delays every later one — mitigate with aggressive timeouts, pool isolation, and per-class queues. - **Hedged requests / request replication**: send redundant requests after a threshold (for example, p95) to cut the tail across replicas and caches (Google's Retwis/fan-out process). --- ## ⚙️ 3. Latency Reduction by Layer ### Application Level (eliminate unnecessary work) - **Do not do synchronous NFS/Disk I/O on the critical path**: anything not strictly necessary should be lazy, asynchronous, or precomputed. - **Caching (with careful invalidation)**: immutable/static data; watch out for stampede (thundering herd) — use a single recompute lock and TTL jitter. - **Batching with a conscious trade-off**: grouping requests raises throughput but adds latency; in low latency, prefer batching over a very short time window (Nagle off, coalescing windows in ms). - **Precomputation and materialization**: compute aggregations offline (event sourcing + projections) instead of on request. - **Serialization**: prefer compact binary formats (Protobuf, Avro, FlatBuffers) over JSON on the hot path; avoid serializing the same structure multiple times. ### RPC / Network Level - Reduce round-trips: combine calls, use connection pipelining, HTTP/2 multiplexing, or gRPC. - **UDP and QUIC**: protocols with 0-RTT and recovery without transport-level head-of-line blocking. - Timeouts and retry budgets with exponential backoff + jitter; never chain infinite retries (amplification). ### Hardware and Runtime Level - **CPU cache hierarchy**: organize data structures for data locality (arrays/AoS vs. SoA, 64-byte cache lines, avoid pointer chasing). - **Branch prediction and prefetching**: hardware predicts branches and prefetches caches; compact "hot" data structures beat generic "rustic" ones. - **NUMA**: on multi-socket servers, pin threads and memory to the same socket (`numactl`) to avoid cross-socket traffic. - **Kernel bypass / modern NICs**: for microsecond latencies, techniques such as busy-spin instead of blocking syscalls, huge pages, and isolated cores (isolcpus + IRQ affinity). - **Memory allocation**: avoid the GC pause path in managed runtimes; prefer short-lived objects (Gen0/Young gen) and object pools for large allocations. ### Distributed Systems Level (consistency × latency) - **PACELC**: in normal operation, *Else* Latency vs. Consistency — strongly consistent replicas pay a round-trip; eventual consistency accepts stale reads at lower latency. - **Sharding**: single-shard queries beat cross-shard queries; choose partition keys that group data accessed together. - **Asynchronous replication** reduces write latency but creates a stale-read window; synchronous replication pays an RTT per write. - **Consensus (Raft/Paxos)** requires a majority — expect a confirmation RTT; deploy in topologies that minimize the physical distance between replicas. --- ## 🧪 4. Six-Step Optimization Protocol 1. **Define the latency SLO per operation** (p50/p99/p99.9 and peak) before optimizing; no target means blind optimization. 2. **Measure with correct telemetry**: distributed tracing (OpenTelemetry), high-resolution timers, reduction of coordinated omission in benchmarks. 3. **Model capacity** with Little's Law/Amdahl to validate numeric hypotheses before refactoring. 4. **Find the real bottleneck** (profile flame graphs, DTrace/perf, per-stage pipeline metrics) — do not guess. 5. **Eliminate the work, then optimize the remaining work** (the fastest optimization is not doing it). 6. **Validate under real load** with A/B or blue/green; latency must be observed continuously in production (synthetics + RUM). --- ## 🔗 Integration with Other Skills - [system-design-scalability](../system-design-scalability/SKILL.md): CAP/PACELC trade-offs, sharding, and distributed caching at scale. - [data-intensive-systems](../../../data/data-intensive-systems/SKILL.md): replicas, partitioning, and queues that influence end-to-end latency. - [lang-python](../../../languages/lang-python/SKILL.md) and [python-performance-parallelism](../python-performance-parallelism/SKILL.md): CPU/GC optimization in interpreted runtimes. - [observability-correlation](../../../mapping/observability-correlation/SKILL.md): instrumentation (traces, metrics) to locate bottlenecks. - [lang-java](../../../languages/lang-java/SKILL.md): JVM tuning (GC, JIT, thread pools) with latency incentives.