| name | performance-profiling-optimization |
| description | Guides systematic performance investigation from symptom to fix — establishing a baseline, profiling the bottleneck, identifying root cause, applying a targeted change, and verifying the improvement with data. Use when someone has a concrete... |
| scenarios | ["Our API response times degraded after the last deploy — help me profile and fix it","Profile this Java service to find memory leaks and CPU hotspots","Application throughput dropped by 40% — systematic performance investigation needed","배포 후 API 응답 시간이 느려졌는데 원인을 찾아줘","메모리 누수와 CPU 병목을 프로파일링해줘"] |
| compatibility | {"recommended":["sequential-thinking","think-tool"],"optional":[],"remote_mcp_note":"sequential-thinking은 베이스라인 측정 → 프로파일링 → 원인 분석 → 수정 → 검증 흐름을 강제합니다. think-tool은 여러 병목 후보 중 우선순위를 결정하는 데 활용됩니다. Claude 설정 → MCP Servers에서 remote SSE 엔드포인트를 추가하세요."} |
Performance Profiling & Optimization
Measure first. Hypothesize second. Profile third. Fix fourth. Verify always.
Never optimize without data. Never guess at bottlenecks.
Investigation Workflow
Step 1 — Establish a Baseline
Before touching anything, capture current state:
- P50 / P95 / P99 latency (not just average)
- Error rate and error types
- Resource utilization: CPU %, heap used/max, GC frequency, thread pool saturation
- Request rate (RPS/TPS) at the time of degradation
Output: a one-line problem statement.
"P99 API latency is 4.2s under 300 RPS; CPU is 40%, heap is 85% of max, GC pauses avg 800ms."
Step 2 — Form a Hypothesis
Map symptoms to candidate causes using the table below. Pick the top 2–3.
| Symptom | Likely Cause |
|---|
| CPU 100%, low latency variance | Hot loop, serialization overhead, regex |
| High P99, low CPU | Lock contention, thread starvation, GC stop-the-world |
| Memory grows then crashes | Memory leak, unbounded cache, large result sets held in memory |
| IO wait high | Slow DB queries, missing index, full table scans |
| Network latency spikes | DNS resolution, connection setup overhead, chatty API calls |
| GC pause > 200ms | Large old-gen, too-frequent allocation, premature promotion |
Step 3 — Profile (tool guidance below)
Target the hypothesis. Don't profile everything at once.
Step 4 — Fix
One change at a time. Document what you changed and why.
Step 5 — Verify
Re-run the same load that reproduced the problem. Compare against baseline.
If the fix holds: document it. If not: return to Step 2.
Profiling Tool Guide
CPU Profiling
Goal: find where CPU cycles are spent — hot methods, tight loops.
- async-profiler (JVM): low-overhead sampling profiler. Produces flame graphs.
./asprof -d 30 -f /tmp/flamegraph.html <pid>
- py-spy (Python): sampling profiler, no code changes needed.
py-spy record -o profile.svg --pid <pid>
- perf + FlameGraph (Linux system-level):
perf record -F 99 -p <pid> -g -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg
- Reading a flame graph: wide boxes = time-consuming. Look for wide plateaus near the top.
Common CPU findings: serialization/deserialization in hot path, regex compilation inside loops, reflection overhead, N+1 computation.
Memory Profiling
Goal: find what's allocated, what's retained, and why GC can't collect it.
Common memory findings: static collections that grow unboundedly, event listeners never removed, large byte arrays for batch processing, connection/stream objects not closed.
IO Profiling
Goal: find slow queries, missing indexes, excessive disk reads.
- Slow query log (MySQL/PostgreSQL):
SET log_min_duration_statement = 100;
SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 20;
- EXPLAIN ANALYZE: run on any query taking > 50ms.
- strace (Linux): trace syscalls for a process.
strace -p <pid> -e trace=read,write,open,close -T 2>&1 | head -100
- iotop: identify processes doing most IO.
Network Profiling
Goal: find latency between services, connection overhead, chattiness.
Common Bottleneck Patterns
N+1 Query
Signature: DB query count scales linearly with result set size. Latency low per query, high in aggregate.
Fix: batch load with IN (...), use JOIN, or add a data loader.
Thread Pool Exhaustion
Signature: requests queue up, latency spikes, thread count at max, CPU low.
Fix: increase pool size (carefully), or reduce work per thread, or use async I/O.
Memory Leak → GC Pressure → Latency Spikes
Signature: heap grows over hours/days, full GC frequency increases, P99 spikes correlate with GC events.
Fix: find the retention root via heap dump; fix the leak; tune GC only as a temporary measure.
Hot Lock / Synchronized Block
Signature: high CPU, many threads in BLOCKED state in thread dump.
Fix: reduce lock scope, use lock-free data structures, partition the lock.
Large Serialization Payload
Signature: high CPU on serialization threads, large response sizes, high memory allocation rate.
Fix: project only needed fields, paginate, use binary formats (protobuf, avro) for internal APIs.
Making the Case for Performance Work
Frame performance work in cost and user impact terms — not engineering elegance.
Template:
"Current P99 latency is X ms. Industry benchmark / SLA target is Y ms.
At our traffic of Z RPS, each 100ms of extra latency costs approximately [conversion rate / user satisfaction / infra cost].
Proposed fix requires [N days]. Expected outcome: P99 drops to W ms, [cost/impact metric] improves by K%."
Escalation Paths
Quick Reference Checklist
Before declaring the investigation done:
Stack coverage note: Tool examples in this skill are JVM (async-profiler, heap dumps, GC logs), Python (py-spy, memory-profiler), and Go (pprof) centric. For Node.js: use clinic.js or --inspect with Chrome DevTools. For Rust: use cargo flamegraph or perf. The investigation workflow (measure → hypothesize → profile → fix → verify) applies to all stacks regardless of tooling.
