| name | cost-anomaly-diagnosis |
| description | Systematic methodology for investigating cost spikes in production LLM pipelines, identifying root causes (concurrent dispatch, weak guards), quantifying blast radius, and designing atomic single-flight fixes. |
| trigger | When a call costs N× more than expected, or cost metrics show unexplained spikes across a time window. |
| author | hello |
| source_sessions | ["hello_hello's Organization_default_4aef2bde-b28d-4d40-aa8c-b54b0f565741"] |
| contributors | ["hello"] |
| version | 1 |
| created_by_agent | claude_code |
| created_at | "2026-06-13T19:10:47.644Z" |
| updated_at | "2026-06-13T19:10:47.644Z" |
Cost Anomaly Diagnosis & Single-Flight Fix Pattern
When to use
A user reports:
- A single call's cost is 10x+ the expected baseline (e.g., $0.165 vs ~$0.01)
- Cost metrics spike unexpectedly across a time window
- Cost variance is high for similar operations
- They suspect duplicate/concurrent work is driving spend
Workflow
1. Isolate the call and count concurrent runs
Get the call ID, timestamp, and operation name (e.g., analyze-human-portion). Query prod logs for all invocations of that operation against that call in a tight window (typically 30s for webhook/retry races, 5m for cron races).
Typical signature: 5–7 independent dispatchers (webhook, sync-calls-oauth, scheduled jobs, retries, force-reanalyze) all read a NULL guard (e.g., analyzed_at IS NULL) within the window.
2. Measure cost per run and compute the multiplier
From logs/RPC data, extract the cost of a single run. Divide actual cost by baseline to get multiplier (e.g., 13×). The multiplier typically = (concurrent run count) − 1 + margin for retries.
3. Quantify blast radius
Determine the time range: typically 7d, or since the last deploy. Query the source table (e.g., calls) for all rows matching the condition that triggered concurrent runs (e.g., has_human_transfer = true AND analyzed_at IS NULL). Count affected calls and compute cumulative cost impact. Express as percentage of 7d category spend.
Example: 66 calls over 7d with ~13× runcount on each = ~$28 of $43 7d human-portion spend (66%).
4. Diagnose the guard failure
Identify what was supposed to prevent concurrency (e.g., WHERE analyzed_at IS NULL). Pinpoint the failure:
- Weak check-then-act: multiple readers all see NULL, write concurrently → no mutual exclusion.
- No persistence: check doesn't cross a database transaction boundary.
- Timeout/retry collision: re-read happens before first write completes.
This is a gap in dispatch design, not a bug in one PR.
5. Design the atomic fix
Introduce atomic single-flight at the database layer:
- Add a claim column:
calls.human_portion_claimed_at TIMESTAMP.
- Create an RPC returning early if the column is set:
try_claim_human_portion() (mirrors try_claim_alert_slot pattern).
