| name | optimization-baseline-fidelity |
| description | Use BEFORE any cost/latency optimization. A/B control MUST be deployed prod config; gate code-start on stated $X/mo savings; reject forced-OFF controls. |
Optimization Baseline Fidelity
Purpose
Prevent the recurring failure mode where a mechanism is built, demonstrated to "work" in a controlled A/B, and shipped — but produces no real production savings because the A/B control was a configuration that does not exist in stable prod.
When to invoke
Invoke this skill whenever the work involves:
- Adding or changing a cache (per-user, per-campaign, shared, system-prompt, etc.)
- Batching, request coalescing, or N+1 reduction
- Model swap, downgrading tier, or routing change
- Prompt slimming or context trimming
- Dedup or idempotency layer
- Any "fall-through" or "opportunistic" mechanism that lives behind an existing primary
Required workflow
Before writing the first line of code:
-
Quantify the addressable slice first. State the target as a percentage of the measured bill/metric, using real data already on hand (census, billing export, traces, telemetry). If the slice the optimization can actually touch is small after accounting for what's already deployed, STOP — do not build.
-
The A/B control arm MUST be the currently-deployed production config, never "off" / "do nothing" / "uncached" / a hand-picked config that maximizes apparent benefit. If another optimization already wins the same traffic in prod (e.g. an existing cache, implicit caching), the control must include it.
-
A measurement run in a config that does not exist in prod is NOT evidence. Forcing a competing feature OFF to make yours look effective produces a preview-path number, not a prod-savings number. Label it as such or discard it.
-
Gate code-start on a stated incremental-savings-over-baseline target. Write "this saves $X/mo vs the deployed config" as the success criterion before the first commit. If you cannot state it, the premise is unvalidated — STOP.
-
For a fall-through / opportunistic mechanism, compute when it actually fires in prod before building. A fall-through behind an already-warm primary is structurally dead.
Why this rule exists
A shared system/tools Gemini cache was built and driven to a near-merged PR whose only savings land when the per-campaign cache is cold (a sliver of prod). The feature was "proved" with a 74.6% reduction measured with the per-campaign cache forced OFF — an isolated control that does not exist in stable prod. 43 tasks of mechanism-correctness, 0 tasks of marginal-$-vs-baseline. The feature also excluded the 89% test/CI cost center by design, so it was aimed away from the money.
The failure was confusing "the mechanism works" with "the bill drops." The deployed-config control is the fix.
Red flags
- "The mechanism is correct, the savings will follow" (without a stated $X/mo)
- A/B comparison against "no cache" when a cache is already deployed
- "Falls through when the primary is cold" without showing what fraction of prod traffic sees a cold primary
- Targeting a 1% slice of the cost center when 89% is elsewhere
Output expectation
State the success criterion ("this saves $X/mo vs the deployed config") before code, and produce measurement artifacts that compare against the deployed config — not a forced-OFF control.
