| name | ramp-and-verify |
| description | Use after a route decision exists and a candidate model must take live traffic safely — "ramp this route", "move 25% of traffic to the new model", "did the route change regress anything", "roll this back", "prove the savings are real". Pre-ramp stability gates, a staged traffic ladder through the Understudy gateway dial, routed-vs-passthrough verification from captures at each step, and explicit rollback triggers. |
| metadata | {"understudy":{"mode":"production","safety":"approval-required","cli_required":true}} |
Ramp and Verify
Every successful journey through this library ends the same way: a candidate
won on a frozen eval and a route decision says ship it. This worker owns the
last mile — taking that candidate from 0% to 100% of live traffic on the
gateway dial without trusting a single lab number, and producing the measured
before/after that makes any savings statement honest. It starts where
../compare-model-sweep/SKILL.md and
../use-understudy-gateway/SKILL.md
stop.
Resolve CLI
Prefer the installed understudy binary. If it is unavailable inside a repo
checkout, run through the package script:
npm run build
node dist/bin.js status --json
Safety Gates
- Every traffic change is an explicit, approved action. Name the workload,
the model id, the old and new percentage, and get approval in the current
thread before each
routes set. Never ramp two tiers in one step.
- Disclose capture-on-by-default. Setting a model route enables request
capture for that workload unless capture is explicitly disabled — tell the
developer this before the first route write and confirm the capture setting
they want. The hosted capture behavior is documented at
docs.understudylabs.com/concepts/capture.
- Snapshot before every change. The CLI writes a route snapshot before
route writes; verify it exists so
routes rollback has something to restore.
- No savings claim without a claim packet. Measured routed-vs-passthrough
deltas go into the evidence contract from
../optimize-workload/SKILL.md (claim.json);
a tier that "looks cheaper" is not a claim.
- Captures contain raw request/response bodies. Read them through the CLI's
redacted, metadata-first views; full payload export is opt-in, file-only,
and approval-gated.
Pre-ramp gates (all must pass)
- Frozen-eval verdict exists — a sweep/optimization result on frozen
splits with the incumbent baseline recorded, or a
pass launch verdict
from ../simulate-before-launch/SKILL.md
for the specific change being ramped (its contract axes — schema validity,
tool-call validity — are the same ones step 3 verifies from captures).
- Repeat-replay stability. Re-run the candidate on the frozen rows in
incremental batches. Three repeats may serve as a plumbing smoke, but size N
for the agreed instability tolerance and increase it when results sit near
the threshold or vary across batches. Bucket each row: all-repeats-match /
some / none (the procedure lives in
../compare-trajectories/SKILL.md).
Rows that never repeat are stochastic pockets — assign each a disposition:
fallback (route to incumbent), shadow (observe, don't serve), or
requires-fresh-traffic. Stop if >~5% of rows are unstable.
- Fallback exists. The non-routed remainder passes through to the
incumbent by construction; confirm the workload's passthrough path works
(
gateway probe) before sending anything to the candidate.
Desktop app export packets under ~/.understudy/exports/ are admissible for
gate 1 once verified — shape, sibling-JSONL hash, split identity, and cost
basis all check out. The admission checklist and verification transcript live
in reference.md.
The ladder
Default tiers: 5% → 25% → 100% (note: routes set defaults --traffic-pct
to 10 when not specified, so pass --traffic-pct 5 explicitly for the first
tier). The dial's hosted split semantics are documented at
docs.understudylabs.com/concepts/routing.
At each tier:
-
understudy routes set <workload> --project <p> --model-id <id> --traffic-pct <pct> — after approval.
-
Soak. Hold the tier for an agreed window (hours at 5%, a day at 25%)
while traffic accumulates.
-
Verify from reporting + captures. Check the reporting endpoints first
for a fast aggregate view:
GET .../provider-health?window=<soak-window>
GET .../status?window=<soak-window>
(docs)
Use
../check-routing-health/SKILL.md
for the full diagnostic flow. Then pull the window's captures
(metadata-first) and compare routed vs passthrough on the same period:
error/status-code rate, latency distribution, schema/parse validity of
outputs, and token/cost per call. The routed cohort must hold the lab
quality bar on whatever the workload's validator can score from captures.
-
Check determinism on a decision-sized stratified set. Re-run routed rows
against the candidate, expanding the set when variance, rare strata, or
failures remain uncertain. If repeats disagree materially more than the
pre-ramp measurement, stop the ramp and re-diagnose. Follow
../capture-evidence/references/evaluation-evidence-gates.md.
-
Advance or roll back. Advance only when the tier window is clean.
Rollback triggers — any one fires understudy routes rollback (or
routes clear) immediately: error rate or schema-validity regression vs
passthrough, tail-latency regression beyond the agreed bound, instability
above the pre-ramp measurement, or the developer's call. After a rollback,
do not re-ramp until the root cause is fixed and the pre-ramp gates pass
again on a fresh eval.
After 100%
Keep the incumbent reachable (snapshot retained, rollback tested) for an
agreed observation window. Then assemble the before/after: incumbent baseline
vs routed steady-state on quality-from-captures, latency, and cost/call —
into the claim packet. That packet, not the ramp log, is what any cost or
quality statement cites. It inherits the claim-packet rigor fields from
../optimize-workload/SKILL.md: the
trivial-agent floor (the claim is invalid if a do-nothing agent also clears
the quality bar) and bootstrap confidence intervals on the before/after
quality delta — an overlap means "no measured regression", never "measured
improvement" (see ../../docs/benchmark-rigor.md).
Output Standard
End with: the workload/project/model ids and the tier history (pct, window,
verdict per tier); the pre-ramp gate results (stability fraction, disposition
counts); per-tier routed-vs-passthrough table (errors, latency, validity,
cost/call); any rollbacks and their trigger; the current route state and
snapshot path; result type (live); and the claim-packet status with the one
recommended next action.
References
