optimization-baseline-fidelity

name	optimization-baseline-fidelity
description	Use BEFORE any cost/latency optimization. A/B control MUST be deployed prod config; gate code-start on stated $X/mo savings; reject forced-OFF controls.

Optimization Baseline Fidelity

Purpose

Prevent the recurring failure mode where a mechanism is built, demonstrated to "work" in a controlled A/B, and shipped — but produces no real production savings because the A/B control was a configuration that does not exist in stable prod.

When to invoke

Invoke this skill whenever the work involves:

• Adding or changing a cache (per-user, per-campaign, shared, system-prompt, etc.)
• Batching, request coalescing, or N+1 reduction
• Model swap, downgrading tier, or routing change
• Prompt slimming or context trimming
• Dedup or idempotency layer
• Any "fall-through" or "opportunistic" mechanism that lives behind an existing primary

Required workflow

Before writing the first line of code:

1. Quantify the addressable slice first. State the target as a percentage of the measured bill/metric, using real data already on hand (census, billing export, traces, telemetry). If the slice the optimization can actually touch is small after accounting for what's already deployed, STOP — do not build.

2. The A/B control arm MUST be the currently-deployed production config, never "off" / "do nothing" / "uncached" / a hand-picked config that maximizes apparent benefit. If another optimization already wins the same traffic in prod (e.g. an existing cache, implicit caching), the control must include it.

3. A measurement run in a config that does not exist in prod is NOT evidence. Forcing a competing feature OFF to make yours look effective produces a preview-path number, not a prod-savings number. Label it as such or discard it.

4. Gate code-start on a stated incremental-savings-over-baseline target. Write "this saves $X/mo vs the deployed config" as the success criterion before the first commit. If you cannot state it, the premise is unvalidated — STOP.

5. For a fall-through / opportunistic mechanism, compute when it actually fires in prod before building. A fall-through behind an already-warm primary is structurally dead.

Why this rule exists

A shared system/tools Gemini cache was built and driven to a near-merged PR whose only savings land when the per-campaign cache is cold (a sliver of prod). The feature was "proved" with a 74.6% reduction measured with the per-campaign cache forced OFF — an isolated control that does not exist in stable prod. 43 tasks of mechanism-correctness, 0 tasks of marginal-$-vs-baseline. The feature also excluded the 89% test/CI cost center by design, so it was aimed away from the money.

The failure was confusing "the mechanism works" with "the bill drops." The deployed-config control is the fix.

Red flags

• "The mechanism is correct, the savings will follow" (without a stated $X/mo)
• A/B comparison against "no cache" when a cache is already deployed
• "Falls through when the primary is cold" without showing what fraction of prod traffic sees a cold primary
• Targeting a 1% slice of the cost center when 89% is elsewhere

Output expectation

State the success criterion ("this saves $X/mo vs the deployed config") before code, and produce measurement artifacts that compare against the deployed config — not a forced-OFF control.