| name | inspector-performance |
| description | Inspector performance across the whole stack — single-worker backend (SQLite / caching / bucket-I/O / hot paths), frontend (bundle / render / wire-shape), and the live-stack runtime probing that produces the evidence (Playwright/Chrome MCP + Flask/Vite log correlation). The cost model, the measurement playbook, and the cautionary patterns. |
inspector-performance
Performance skill for the Inspector. Standalone — references split by layer. This replaces the old performance.md rule: same intent (a public web app on one worker, where per-request waste compounds across users), more depth, plus the runtime-probing methodology folded in.
The cost model (internalize this before optimizing anything)
Single-worker gunicorn -w 1, 16 gthreads, 2 vCPU / 18 GB HF Space. Consequences that drive every decision:
- Pure-CPU work serializes across all users. A 1 s CPU spike (a big
orjson dump, a pydantic catalog rebuild, an un-cached classify) blocks all 16 gthreads. Plan for sub-100 ms on per-request hot paths. The GIL means threads only help I/O-bound work (bucket reads, SSL recv release the GIL); CPU work does not parallelize.
- Bucket reads are the slowest layer. FUSE mount in prod;
hffs.cat_file fallback is ~50–500× slower when unmounted. Treat each bucket read as expensive; fan out independent reads with ThreadPoolExecutor.
- SQLite is the source of truth, one serialized writer (
connection._WRITE_LOCK), WAL readers concurrent. State transitions block on a synchronous full-DB upload to the bucket before acking (durable_transaction).
- Aggregate is the real number. A 100 ms regression × N concurrent users × M reads/session compounds. The arithmetic that matters is "× N", not the single-call wall-clock.
What changed vs the old rule (don't trust stale framing)
- The old
performance.md pointed at the inspector-runtime-prober agent as the probing entry point. That agent is gone — performance-expert owns probing now, and the methodology lives in references/probing.md.
- There is no audio prefetch worker and no GC sweeper (both removed). Don't reason about TTL purge or fetch-warming as live costs.
Hot paths (where the time actually goes)
| Path | Dominant layer | Cold | Warm |
|---|
| Dashboard / public catalog load (the anonymous concurrent path) | Python CPU (pydantic catalog rebuild + orjson dump) | ~38 ms, projected ~700 ms @100× deliveries | dict handback (db_seq-keyed byte cache) |
Segments validate (/api/seg/validate) | cold = bucket I/O (4 parallel reads); warm = CPU classify | 4-read fan-out | cache miss only after save |
Peaks (/api/seg/peaks) | cold = bucket I/O (≤114 gzip shards, 8-way fan-out); warm = dict lookup | ~45 ms cold (mounted) | pre-serialized orjson bytes |
Save (save_seg_data) | bucket I/O (4–5 serialized writes/reads) | — | — |
Audio play (audio-proxy) | network / OS sendfile, ~0 Python CPU on the mount path | — | — |
| State transition | bucket I/O (synchronous full-DB upload, CAS-guarded) | — | — |
Details + file:line in references/backend-db.md.
Reference index
| Reference | When to read | Key files |
|---|
references/backend-db.md | Compute placement (write vs read), the full cache inventory + invalidation, I/O fan-out + concurrency, save/validate/catalog cost, the two unbounded caches, the bench/drift measurement playbook, local-vs-HF | services/storage/cache.py, services/db/{sync,connection}.py, services/segments/save.py, services/validation/__init__.py, routes/segments/peaks.py, scripts/bench_storage.py |
references/frontend.md | FE critical path, bundle/chunk splitting, the two-layer tab gate, lazy panels, wire-shape, $effect hygiene, audio preload audits, the CSS/JS bundle regrets | frontend/vite.config.ts, src/main.ts, src/App.svelte, tabs/*/stores, lib/refs/quran-refs.ts, lib/utils/peaks-decode.ts |
references/probing.md | Driving the live stack: bring-up commands, exact Playwright/Chrome MCP tool calls + browser_evaluate recipes, browser↔Flask-log correlation, ranked-summary discipline, local-vs-HF for probing | — |
Method (the non-negotiables)
- Measure before guessing. cProfile /
perf_counter markers / bench_storage.py. Numbers > intuition.
- Cold and warm are different problems. Cold = post-save / first-load (bucket I/O). Warm = subsequent interaction (CPU). Optimize them separately; never report one as the other.
- Isolate Python CPU from wall-clock. CDN/HTTPS/disk variance masks CPU. Profile the service function with caches pre-primed (no bucket read) to see the CPU floor.
- Idle-system rule. Take numbers with loadavg < 1; on one worker, background load skews everything.
- Any change to a perf-sensitive path ships with: a drift check (byte-equivalent vs ground-truth snapshot), before/after numbers, and the idle-system caveat. Compute-placement / cache / parallelism / persisted-field-writer changes especially.
Cautionary patterns (the "had we done X first" core)
- Cache invalidation that isn't keyed on a version is hand-maintained and fragile. The DB caches self-invalidate on
db_seq; the bucket-file caches don't, so their invalidation is a choreography in save.py that's easy to get subtly wrong (the auto-split "only when uid set changes" gate is exactly this). New cache → entry in cache.py + an explicit invalidation hook; never an ad-hoc module global.
- Two caches are unbounded (
_PEAKS_CACHE, _AUDIO_CACHE_STATUS) — they grow per (reciter × url) for the process lifetime. Don't add a third; bound new caches (LRU / explicit eviction).
- Recompute-on-read of a pure-over-stable-input value is waste. Persist it at write-time behind one source-of-truth helper, drift-gated (qalqala, boundary_adj). Asymmetric save-vs-read ownership is a smell.
- Sequential bucket reads in a hot path are a code smell — check whether each call truly depends on the prior one; fan out if not.
- Wire-shape bloat compounds (~25 B/row × thousands × many users). Strip unused fields aggressively; keep payloads slim (slim int8 peaks,
limit-capped catalog).
- Eager work the user may never need — a tab's
onMount fetch behind a hidden=-only gate, a preload="metadata" audio element mounted in a panel, a heavy dep in the main bundle. Defer mount (not just visibility), lazy-load panels, dynamic-import-gate heavy chunks.
Relationship to docs/reference/ (don't create drift)
This skill is a performance lens + practices, not a second copy of subsystem knowledge. The what-is ground truth lives in docs/reference/ and wins on conflict:
- Cache-module mechanics, the invalidation rule, the route map →
docs/reference/architecture.md
- SQLite substrate, repos,
sync.py, db_seq → docs/reference/database.md
- Validation engine, persisted classifier fields, the bench/drift harness →
docs/reference/validation.md
- FE SPA shell, stores, lazy-mount →
docs/reference/frontend.md
What this skill owns (no reference doc does): the cost model, the consolidated cache inventory (perf lens — bounded/unbounded, hit/miss), the hot-path cost map, the measurement/profiling playbook, and the runtime-probing methodology. When cache.py / a hot path / a persisted-field writer changes, update the owning reference doc AND this skill's lens. Don't restate reference mechanics here beyond the one-line anchor needed to make a cost point.
What this skill deliberately doesn't cover
Audio-subsystem internals (see the inspector-audio skill — performance-expert delegates audio-domain reasoning to audio-expert). Deployment/image build (see docs/reference/config-deploy.md). DB schema/migration mechanics beyond their perf characteristics (see docs/reference/database.md).
