performance-engineering

name	performance-engineering
description	Use when the user says "this is too slow", "how do I optimize this", "help me profile this", "our p99 is too high", "we're hitting database limits", or discusses latency, throughput, memory usage, or CPU profiling. Provides systematic performance engineering methodology.

Performance Engineering

Measure before you optimize. The most expensive optimization is one that targets the wrong bottleneck. Performance engineering is a discipline of instrumentation, profiling, and hypothesis-driven experimentation — not intuition-based micro-optimization.

When to Activate

• A service has unexpectedly high latency or low throughput
• CPU or memory usage is growing unexpectedly
• A database is under high load
• Preparing for a load test or capacity planning
• Reviewing code for performance implications

The Performance Engineering Loop

1. DEFINE   → set a specific, measurable target (p99 < 200ms at 1000 RPS)
2. MEASURE  → establish a baseline with real data, real load
3. PROFILE  → find the actual bottleneck (not the suspected one)
4. OPTIMIZE → change exactly one thing
5. MEASURE  → quantify the improvement against the baseline
6. REPEAT   → until the target is met or the cost of further optimization exceeds the benefit

Never skip step 1. "Make it faster" is not a success criterion. "Reduce p99 latency from 800ms to 200ms at peak load" is.

Latency Targets and SLOs

Define SLOs before performance work:

• p50 (median): typical user experience
• p95: most users' worst case
• p99: tail latency — often what SLAs are written against
• p99.9: extreme tail — relevant for high-volume systems where 0.1% = thousands of users

Tail latency is disproportionately expensive to fix — the last 10% of latency reduction often requires 10x the engineering effort. Define acceptable thresholds and stop there.

Finding the Bottleneck

The Four Bottleneck Types

1. CPU-bound: the process is spending time computing. Profile call stacks to find hot functions.
2. I/O-bound: the process is waiting for disk or network. Reduce I/O operations, use async/concurrency, cache.
3. Memory-bound: GC pressure, swapping to disk, or cache thrashing. Profile heap allocation.
4. Lock-bound: threads waiting on contended locks. Profile lock contention, reduce critical sections.

Confirm which type before optimizing. CPU optimizations do nothing for I/O-bound processes.

Profiling Techniques

CPU profiling: sampling profiler captures call stack snapshots at intervals. Find which functions consume the most CPU time.

# Python
python -m cProfile -o output.prof my_script.py
snakeviz output.prof

# Node.js
node --prof app.js && node --prof-process isolate-*.log

# Go
import _ "net/http/pprof"
go tool pprof http://localhost:6060/debug/pprof/profile

Memory profiling: track allocations over time. Look for monotonically growing objects.

Distributed tracing: for services, trace requests end-to-end. Find which span dominates latency. OpenTelemetry is the standard instrumentation format.

Database query profiling: the most common bottleneck.

EXPLAIN ANALYZE SELECT ...;  -- PostgreSQL: shows actual vs estimated rows, timing
SHOW PROFILES;               -- MySQL

Look for: sequential scans on large tables, nested loop joins with high row estimates, sorts without indexes.

Common Performance Problems and Fixes

N+1 Queries

Symptom: for a list of N items, the code issues N additional queries.

# N+1: 1 query for orders, then 1 query per order for user
orders = Order.all()
for order in orders:
    print(order.user.name)  # lazy load triggers per-iteration query

# Fixed: eager load
orders = Order.includes(:user).all()

Detection: enable query logging and count queries per request. A request that issues 50+ queries is almost certainly N+1.

Missing Database Index

A full table scan on a large table is the single most common performance problem. Every column used in a WHERE, JOIN, or ORDER BY clause should be indexed (unless the table is small or the query is rare).

-- Find slow queries
SELECT query, calls, mean_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 20;

-- Find tables with sequential scans
SELECT relname, seq_scan, idx_scan
FROM pg_stat_user_tables
WHERE seq_scan > 0
ORDER BY seq_scan DESC;

Unnecessary Serialization

JSON serialization/deserialization is expensive at high volume. If a service serializes a large object, transmits it, and the consumer immediately deserializes it to access one field — the work is wasted.

Fix: transmit only the data the consumer needs. Apply pagination and field selection to APIs.

Cache-Unfriendly Access Patterns

Caching reduces latency only if the cache hit rate is high. A cache with a 50% hit rate reduces latency by 50% on average but doubles the write amplification.

Measure cache hit rate. If it's below 80%, investigate:

• TTL is too short
• Key space is too large (too many unique keys)
• Eviction policy is inappropriate for the access pattern

Blocking I/O in Async Code

In async/event-loop frameworks (Node.js, Python asyncio), a single blocking call blocks the entire event loop.

# Blocks the event loop — all other requests wait
time.sleep(5)
requests.get(url)  # synchronous HTTP in async context

# Fixed: use async I/O
await asyncio.sleep(5)
await aiohttp_session.get(url)

Load Testing

Load test before you need the capacity, not during an incident.

Define the scenario: realistic request mix, realistic data size, realistic user behavior (not just hitting one endpoint at maximum rate).

Tools: k6, Locust, Artillery, JMeter.

Ramp pattern: start at 10% of target load, ramp to 100% over 10 minutes, sustain for 20 minutes, ramp down. Watch for:

• Latency increase as load increases (find the knee of the curve)
• Error rate increase (find the overload point)
• Memory growth that does not stabilize (memory leak under load)

Capacity planning: if the system handles 1000 RPS at p99 < 200ms with 60% CPU utilization, you have ~40% headroom. Plan to scale before you hit 80% utilization.

Gotchas

1. Optimizing before measuring: the code you think is slow is rarely the bottleneck. The profiler will tell you what actually is.

2. Benchmarking in the wrong environment: a benchmark on a laptop with a local database will produce numbers that do not reflect production. Benchmark on production-equivalent hardware with production-equivalent data.

3. Caching without measuring cache hit rate: a cache that is never hit makes the system more complex and no faster. Measure hit rate from day one.

4. Ignoring tail latency: p50 optimization often makes p99 worse. Measure and target the right percentile.

5. Premature index addition: indexes speed reads and slow writes. Adding indexes to every column increases write latency and storage. Index selectively, based on query analysis.

6. Micro-benchmarks that don't reflect real usage: a function that is 10x faster in a microbenchmark may be called once per request — irrelevant. Find the functions on the hot path.

7. Thread pool exhaustion as the actual bottleneck: a service that appears CPU-bound may actually be waiting on a thread pool that is too small. Check thread pool metrics before scaling CPU.

Integration

• system-design — scalability decisions made at design time determine the performance ceiling
• debugging-methodology — performance regressions are bugs; apply the debugging loop
• incident-response — latency spikes in production are incidents; measure, hypothesize, fix
• testing-strategy — performance regression tests catch slowdowns before they reach production

References

• Profiler Guide — annotated profiling scripts for Python, Node.js, and Go

Skill Metadata

Created: 2026-04-10 Version: 1.0.0