loom-performance-testing

# Performance Testing ## Overview Validate that a system meets latency, throughput, and stability targets under load, and locate the bottleneck when it doesn't. This file assumes you can write the test code; it focuses on the measurement traps that make load-test numbers lie and the decision criteria for tool/shape selection. ## Core Concepts (read first — these are where numbers go wrong) ### Open vs. closed workload models The single most consequential choice. It determines what your numbers mean. | Model | Load driver | Throughput is… | Overload behavior | Use for | | ---------- | ------------------------------------ | ------------------------ | ------------------------------------------------ | -------------------------------- | | **Closed** | Fixed VUs, each loops request→wait | *Emergent* (backpressure) | Self-throttles: slow server → fewer requests sent | Modeling a fixed client pool | | **Open** | Fixed *arrival rate* (req/s) | *Controlled* (you set it) | Queue grows unbounded; latency explodes | Web traffic, finding breaking pt | ⚠ **Closed models hide overload.** With fixed VUs, when the server slows down each VU sends *fewer* requests, so offered load silently drops. You can't overwhelm the server past what its own latency allows — you measure a moving target, not capacity. Real internet traffic is open (users arrive independently of server health), so **use an arrival-rate executor to find true breaking points.** - k6: closed = `constant-vus`/`ramping-vus`; open = `constant-arrival-rate`/`ramping-arrival-rate`. - Locust is closed-model (users); JMeter thread groups are closed; Gatling `injectOpen`/`injectClosed`; artillery `arrivalRate` is open. - Open executors need `preAllocatedVUs`/`maxVUs` headroom; if k6 warns "insufficient VUs," it *dropped* iterations and your rate was never achieved. ### Coordinated omission — why naive latency numbers lie A closed-model generator that waits for each response before sending the next **stops the clock during a stall.** One 10s stall that should have blocked 10,000 scheduled 1ms requests gets recorded as *one* 10s sample instead of 10,000 samples ≥ their overdue time. Result: p99 looks great while users are timing out. - **Detect:** suspiciously clean tail (p99 ≈ p50) under load that clearly stuttered; throughput dips that don't show in latency percentiles. - **Fix:** measure against an *intended schedule*, not send-time. Use tools that correct it: - `wrk2 -R<rate> --latency` (constant throughput + HdrHistogram correction) — the reference implementation. - k6 arrival-rate executors record each iteration against its scheduled start, so `iteration_duration` is corrected; still prefer open model for tail truth. - HdrHistogram `recordValueWithExpectedInterval(v, interval)` back-fills omitted samples. - **Never** report tail latency from a closed-loop `ab`/single-thread `wrk` run at saturation. ### Percentiles, not averages - Report **p50 / p95 / p99 / max** — never mean alone. Averages hide the tail; the mean can be 40ms while p99 is 3s. - Latency distributions are right-skewed and multi-modal (cache hit vs miss, GC pause). Mean sits in an empty valley between modes. - **p99 is a per-request user probability, not a rare edge case:** a page issuing 100 requests hits its p99 latency on ~63% of loads (1−0.99¹⁰⁰). Tail latency *is* the typical experience for fan-out. - Percentiles **don't average across shards** — never mean the p99s of two load-gen workers. Merge the raw histograms (HdrHistogram) or you get garbage. ### Little's Law — sanity-check every run `L = λ × W` → concurrency = throughput × latency. - Required VUs ≈ target_RPS × avg_response_time_s. Wanting 1000 RPS at 200ms needs ≥200 concurrent in flight. - If achieved `RPS ≈ VUs / (latency + think_time)` doesn't hold, your generator is the limit, not the server. ### Saturation: knee vs. collapse As load rises: throughput climbs, then **plateaus (the knee)** while latency turns hockey-stick. Past the knee, more load buys only more latency and eventually errors/collapse. - **Report the knee** (max sustainable throughput at acceptable latency), not the collapse point. - USE method for the SUT: **U**tilization, **S**aturation (queue depth/run-queue), **E**rrors — per resource (CPU, mem, disk, net, connection pool). ### Warm-up, JIT, GC — discard the transient - **Always discard a warm-up window.** Cold caches, empty connection pools, unfilled DB buffer pool, and JIT (JVM/V8) compilation make the first seconds meaningless. In k6 use a ramp stage; in JMH warm-up iterations; in Gatling a `nothingFor`/ramp. - **GC/JIT cause tail latency**, not throughput loss — stop-the-world pauses show up as p99/max spikes. A soak test surfaces them; a 30s run won't. - JVM needs thousands of iterations to reach steady state (C2 compiler). Benchmark JVM services *after* warm-up or numbers are 2–10× pessimistic. ### Measure the server, not the client - **Run the generator on a separate host** from the SUT — co-located, they fight for CPU/net and you measure interference. - **Load-generator-is-the-bottleneck** is the most common false result. Detect: latency rises but *server* CPU is low while *generator* CPU pegs, or you hit ephemeral-port/FD exhaustion. Symptoms: - Single-threaded generators (`ab`, artillery/Node, locust single-worker) cap ~1 core. - Ephemeral port exhaustion (`netstat` full of TIME_WAIT) → looks like server refusing connections. - FD limit (`ulimit -n`) caps concurrent connections. - **Fix:** multi-core generators (k6, Gatling, wrk, JMeter), distributed mode (locust master/worker, k6 cloud/operator), raise `ulimit -n`, reuse connections. **Always correlate client numbers with server-side metrics** (APM/Prometheus) — divergence means you're measuring the wrong thing. ### Connection reuse & think time - **Keep-alive dominates:** new-connection-per-request adds TCP + TLS handshake (often > the request itself). Most tools reuse per VU by default (k6 does). Test *both* if clients differ (mobile cold-start vs. pooled service). k6: `noConnectionReuse: true` / `noVUConnectionReuse`. - **Think time** models real users (pauses between actions) and, in closed models, sets effective concurrency (Little's Law). Zero think time in a closed model = an unrealistic hammer that mostly measures your keep-alive path. Use randomized think time (not fixed — fixed sleeps create synchronized request waves). ### Realistic data & cardinality - **Hitting one key lies:** repeating `GET /users/1` measures a cache/DB-buffer hit path nobody experiences. Use realistic key distribution — **Zipfian/Pareto** (80/20), not uniform, not constant. - Dataset must exceed cache size, or you benchmark RAM. - Unique-per-request params defeat query-plan/result caches (good for worst case); constant params inflate results. - Match production **payload sizes, cardinality, and content** (unicode, large records). --- ## Load Shapes One canonical k6 config per shape. Swap `constant-vus`→`constant-arrival-rate` (and `ramping-vus`→`ramping-arrival-rate`, `target` = req/s) for the open model when finding limits. | Shape | Goal | Pattern | | -------------- | -------------------------- | ------------------------------- | | **Smoke** | Validate script works | 1–5 VUs, 1 min | | **Load** | Baseline at expected load | Ramp → hold steady → ramp down | | **Stress** | Find breaking point | Step up past expected until fail | | **Spike** | Sudden burst recovery | Jump to N×, hold, drop, recover | | **Soak** | Leaks/degradation | Moderate load for hours | | **Breakpoint** | Max capacity | Linear ramp with no cap | ```javascript // Load: warm-up ramp is mandatory; thresholds abort/mark-fail the run. export const options = { stages: [ { duration: "2m", target: 100 }, // ramp (discard for analysis) { duration: "10m", target: 100 }, // steady state = the measurement { duration: "2m", target: 0 }, // ramp down ], thresholds: { http_req_duration: ["p(95)<500", "p(99)<1000"], http_req_failed: ["rate<0.01"], }, }; ``` ```javascript // Stress (open model — reveals true limit): step arrival rate until errors/latency break. export const options = { scenarios: { stress: { executor: "ramping-arrival-rate", startRate: 50, timeUnit: "1s", preAllocatedVUs: 100, maxVUs: 2000, // headroom or iterations drop silently stages: [ { duration: "2m", target: 200 }, { duration: "2m", target: 400 }, { duration: "2m", target: 800 }, // watch for the knee here { duration: "2m", target: 1600 }, ], }, }, }; ``` ```javascript // Spike: validate autoscaling/cache/recovery, not steady throughput. export const options = { stages: [ { duration: "1m", target: 50 }, // baseline { duration: "10s", target: 1000 }, // spike { duration: "3m", target: 1000 }, // hold — does it shed load or fall over? { duration: "10s", target: 50 }, // drop { duration: "3m", target: 50 }, // recovery — latency must return to baseline ], }; ``` ```javascript // Soak: surfaces leaks, FD/connection exhaustion, GC creep, disk fill. Hours, not minutes. export const options = { vus: 100, duration: "4h", thresholds: { http_req_duration: ["p(99)<1000"], http_req_failed: ["rate<0.01"] }, }; // Watch: RSS growth, GC frequency, pool saturation, p99 drift over time (not the average). ``` --- ## Tools ### k6 (default for API/protocol load testing) Go-based, multi-core, JS scripting, open+closed executors, built-in thresholds. Best default. ```javascript import http from "k6/http"; import { check, sleep } from "k6"; import { Rate, Trend } from "k6/metrics"; const errorRate = new Rate("errors"); export const options = { scenarios: { // open model: request rate is what you control, independent of server health api: { executor: "ramping-arrival-rate", startRate: 100, timeUnit: "1s", preAllocatedVUs: 200, maxVUs: 1000, stages: [{ duration: "2m", target: 500 }, { duration: "5m", target: 500 }], }, }, thresholds: { "http_req_duration{expected_response:true}": ["p(95)<500", "p(99)<1000"], http_req_failed: ["rate<0.01"], }, }; export function setup() { const r = http.post(`${__ENV.API_URL}/auth/login`, JSON.stringify({ email: "load@test.com", password: "x" }), { headers: { "Content-Type": "application/json" } }); return { token: r.json("token") }; // runs once; returned data passed to default fn } export default function (data) { const res = http.get(`${__ENV.API_URL}/api/users/${__VU}`, { headers: { Authorization: `Bearer ${data.token}` }, tags: { name: "GetUser" }, // aggregate by name, NOT by dynamic URL (avoids metric cardinality blowup) }); errorRate.add(res.status !== 200); check(res, { "200": (r) => r.status === 200 }); sleep(1); } ``` k6 gotchas: - **Tag dynamic URLs** with a static `tags.name`, or per-URL metrics explode and dashboards choke. - `discardResponseBodies: true` cuts generator memory/CPU when you don't inspect bodies — raises max achievable load. - `check()` failures do **not** fail the run; only `thresholds` do (and `--fail-on-check` / `abortOnFail` on a threshold aborts early). - CLI: `k6 run -e API_URL=... script.js`; `--out experimental-prometheus-rw` or `--out influxdb=...` for Grafana; `k6 run --vus 50 --duration 30s` overrides options. - Extensions (xk6) add gRPC, SQL, browser, Kafka — not in the core binary. ### Locust (Python, closed-model, distributed) Use when scenarios need Python logic/libraries or complex stateful user journeys. ```python from locust import HttpUser, task, between class WebsiteUser(HttpUser): wait_time = between(1, 3) # think time → sets effective concurrency (closed model) def on_start(self): r = self.client.post("/login", json={"username": "u", "password": "p"}) self.token = r.json()["token"] @task(3) # weight: runs 3× as often as weight-1 tasks def browse(self): # name= groups metrics for parameterized URLs (same role as k6 tags.name) self.client.get(f"/api/products/{randint(1,1000)}", name="/api/products/[id]", headers={"Authorization": f"Bearer {self.token}"}) ``` ```bash locust -f lf.py --host=https://api.example.com # web UI locust -f lf.py --headless -u 500 -r 50 -t 5m --host=... # -u users, -r spawn/s locust -f lf.py --master & locust -f lf.py --worker --master-host=IP # distributed: 1 worker/core ```