latency-engineering

# AI Skill: Latency Engineering This skill guides the artificial intelligence to analyze, model, and reduce end-to-end latency in software systems, from theoretical foundations (queuing theory) to low-level implementation techniques, based on the work *Latency: Reduce delay in software systems* (Pekka Enberg, Manning). --- ## 📏 1. Fundamental Laws of Latency ### Little's Law Connects latency (L), throughput (X), and concurrency (N): **N = L × X** - Use it to size connection pools, thread pools, and buffers: at 1,000 req/s and a target latency of 50 ms, the system needs roughly 50 concurrent requests. - If throughput saturates and N grows without bound, all extra capacity goes into the queue (latency explodes). - Mind the assumptions: it holds only in steady state; unstable systems violate the theorem's premise. ### Amdahl's Law Speedup limit from parallelization: **Speedup = 1 / (F + (1 − F)/N)**, where F is the serial fraction. - The serial fraction is the watershed: even with infinite cores, maximum speedup is 1/F. - Use it to estimate diminishing returns from parallelization before investing in more hardware. ### Queuing Theory & Utilization - Latency grows **non-linearly** as utilization (ρ) approaches 1 (M/M/1 queue: latency ∝ 1/(1−ρ)). - Rule of thumb: keep latency-critical services below roughly 70–80% utilization; above that, p99 grows exponentially. --- ## 📊 2. Tail Latency and Coordinated Omission - **Percentiles, not averages**: Measure and optimize p99/p99.9, because at scale a single user makes few requests and will be affected by the tail. - **Distributed systems amplify the tail**: a pipeline of 10 services with a 100 ms p99 can have a far worse end-to-end tail latency (fan-out multiplies the chance of a slow path). - **Coordinated Omission (Gil Tene)**: load generators that synchronize requests (closed-loop) hide stall periods and drastically underestimate the tail. Fix it by reporting the latencies of requests that *should* have been sent during the stall. - **Head-of-line blocking**: one slow request in the queue (even single-threaded, as in Node.js, or in common framework filters) delays every later one — mitigate with aggressive timeouts, pool isolation, and per-class queues. - **Hedged requests / request replication**: send redundant requests after a threshold (for example, p95) to cut the tail across replicas and caches (Google's Retwis/fan-out process). --- ## ⚙️ 3. Latency Reduction by Layer ### Application Level (eliminate unnecessary work) - **Do not do synchronous NFS/Disk I/O on the critical path**: anything not strictly necessary should be lazy, asynchronous, or precomputed. - **Caching (with careful invalidation)**: immutable/static data; watch out for stampede (thundering herd) — use a single recompute lock and TTL jitter. - **Batching with a conscious trade-off**: grouping requests raises throughput but adds latency; in low latency, prefer batching over a very short time window (Nagle off, coalescing windows in ms). - **Precomputation and materialization**: compute aggregations offline (event sourcing + projections) instead of on request. - **Serialization**: prefer compact binary formats (Protobuf, Avro, FlatBuffers) over JSON on the hot path; avoid serializing the same structure multiple times. ### RPC / Network Level - Reduce round-trips: combine calls, use connection pipelining, HTTP/2 multiplexing, or gRPC. - **UDP and QUIC**: protocols with 0-RTT and recovery without transport-level head-of-line blocking. - Timeouts and retry budgets with exponential backoff + jitter; never chain infinite retries (amplification). ### Hardware and Runtime Level - **CPU cache hierarchy**: organize data structures for data locality (arrays/AoS vs. SoA, 64-byte cache lines, avoid pointer chasing). - **Branch prediction and prefetching**: hardware predicts branches and prefetches caches; compact "hot" data structures beat generic "rustic" ones. - **NUMA**: on multi-socket servers, pin threads and memory to the same socket (`numactl`) to avoid cross-socket traffic. - **Kernel bypass / modern NICs**: for microsecond latencies, techniques such as busy-spin instead of blocking syscalls, huge pages, and isolated cores (isolcpus + IRQ affinity). - **Memory allocation**: avoid the GC pause path in managed runtimes; prefer short-lived objects (Gen0/Young gen) and object pools for large allocations. ### Distributed Systems Level (consistency × latency) - **PACELC**: in normal operation, *Else* Latency vs. Consistency — strongly consistent replicas pay a round-trip; eventual consistency accepts stale reads at lower latency. - **Sharding**: single-shard queries beat cross-shard queries; choose partition keys that group data accessed together. - **Asynchronous replication** reduces write latency but creates a stale-read window; synchronous replication pays an RTT per write. - **Consensus (Raft/Paxos)** requires a majority — expect a confirmation RTT; deploy in topologies that minimize the physical distance between replicas. --- ## 🧪 4. Six-Step Optimization Protocol 1. **Define the latency SLO per operation** (p50/p99/p99.9 and peak) before optimizing; no target means blind optimization. 2. **Measure with correct telemetry**: distributed tracing (OpenTelemetry), high-resolution timers, reduction of coordinated omission in benchmarks. 3. **Model capacity** with Little's Law/Amdahl to validate numeric hypotheses before refactoring. 4. **Find the real bottleneck** (profile flame graphs, DTrace/perf, per-stage pipeline metrics) — do not guess. 5. **Eliminate the work, then optimize the remaining work** (the fastest optimization is not doing it). 6. **Validate under real load** with A/B or blue/green; latency must be observed continuously in production (synthetics + RUM). --- ## 🧱 5. Low-Latency C++ Implementation (Ghosh) Theory meets code when building a latency-critical system from scratch (electronic trading, market data). The pattern is a **pipeline of threads joined by lock-free queues**. ### Vocabulary and measurement - Distinguish **latency-sensitive** (improves as latency falls) from **latency-critical** (fails above a threshold). Track **mean, median, peak and jitter** — trading wants low mean *and* low variance. - Latency spans: **time-to-first-byte**, **round-trip time**, and **tick-to-trade**. Measure with `rdtsc` (`asm volatile ("rdtsc" : "=a"(lo), "=d"(hi))`) but beware: it is non-portable and CPU frequency varies per core. Accurate measurement needs a tuned host: interrupts disabled, correct NUMA, pinned power and `isolcpus`. ### Hot-path C++ discipline - Storage hierarchy: **registers > stack locals > everything else**. Prefer `const&` for composite parameters, by value for primitives. Avoid `static`/global (poor cache reuse) and `volatile` (kills register caching and reordering) on the hot path. - **Eliminate dynamic allocation** from the hot path: a pre-allocated memory pool (`std::vector<ObjectBlock>` where the object and its `is_free_` flag share a cache line) fixes fragmentation and pointer aliasing. Only the constructor allocates. - **Avoid virtual dispatch, RTTI and `std::function`** on the hot path — they block inlining and devirtualization. Replace runtime callbacks with **CRTP** compile-time polymorphism. A benchmark showed removing a per-character `std::function` logging callback cut ~25,757 → 466 cycles/op. - Branch hints and cache-friendly layout: ```cpp #define LIKELY(x) __builtin_expect(!!(x), 1) #define UNLIKELY(x) __builtin_expect(!!(x), 0) ``` Prefer branchless selection (index by `sideToInt(side)`) over `if` on a hot path; linearize with loop unrolling/inlining. - **Thread affinity**: pin critical threads with `pthread_setaffinity_np` (`CPU_ZERO`/`CPU_SET`); migration and context switches are costly. ### Lock-free building blocks Locks cause context switches and are rejected on the hot path. The canonical SPSC queue pre-allocates storage and splits access to keep critical windows short: ```cpp template <typename T> class LFQueue final { std::vector<T> store_; std::atomic<size_t> next_write_index_{0}; std::atomic<size_t> next_read_index_{0}; std::atomic<size_t> num_elements_{0}; }; ``` Split `getNextToWriteTo()`/`updateWriteIndex()` and `getNextToRead()`/`updateReadIndex()` to avoid long critical windows and large-object copies; delete copy/move. Threads start with variadic perfect-forwarding plus a core id. **This covers SPSC only** — for MPMC, the ABA problem, hazard pointers, RCU, false-sharing mitigation (`alignas`, `std::hardware_destructive_interference_size`), hugepages and kernel bypass (DPDK, Solarflare Onload) consult a dedicated lock-free reference, since they are outside this outline's scope. ### Networking and ordering TCP for order flow (reliability, in-order), **UDP multicast** for market data (avoids ACK/retransmit bandwidth). Set `TCP_NODELAY` (disable Nagle), non-blocking sockets (`O_NONBLOCK`), `SO_TIMESTAMP`; run a TCP server with **edge-triggered `epoll`** (`EPOLLET | EPOLLIN`). Above the wire, enforce strictly incrementing sequence numbers, heartbeats and a logon handshake; market data uses a **snapshot stream + incremental stream synchronizer** that clears the book, subscribes to snapshots, buffers increments and replays from the snapshot's last sequence. Serialize as **binary-packed POD** structs (SBE-like); JSON/protobuf are not appropriate on this path. ### Compiler flags and methodology `-O3 -flto -fwhole-program -march=native -fno-rtti -fno-exceptions`, PGO with `-fprofile-generate`; note the compiler cannot optimize across modules, through pointers, through floats (induction variables) or through virtual/function pointers. Benchmark **isolated** subsystems with many iterations, and inspect **distribution spikes**, not just means — a function with a 90 µs mean and 1,200 µs spikes has a jitter problem. --- ## 🔗 Integration with Other Skills - [system-design-scalability](../system-design-scalability/SKILL.md): CAP/PACELC trade-offs, sharding, and distributed caching at scale. - [lang-cpp](../../../languages/lang-cpp/SKILL.md): the language constructs (move semantics, RAII, templates) the hot path relies on. - [data-intensive-systems](../../../data/data-intensive-systems/SKILL.md): replicas, partitioning, and queues that influence end-to-end latency. - [lang-python](../../../languages/lang-python/SKILL.md) and [python-performance-parallelism](../python-performance-parallelism/SKILL.md): CPU/GC optimization in interpreted runtimes. - [observability-correlation](../../../mapping/observability-correlation/SKILL.md): instrumentation (traces, metrics) to locate bottlenecks. - [lang-java](../../../languages/lang-java/SKILL.md): JVM tuning (GC, JIT, thread pools) with latency incentives.