Everything else — the artifact contract, measured baseline, workload-specific
decision contract, evidence-driven intervention choice, and stronger RL gates —
is shared.
Default to the path with the highest expected progress toward the multi-objective
decision under hard constraints, not the cheapest rung. State the expected
quality, latency, safety, time, and spend tradeoffs; follow
../understudy/reference.md → Outcome-first spend
posture. Launching a named bounded run plan authorizes its declared provider
calls, uploads, hosted jobs, tool calls, evaluation, receipts, and cleanup. Do
not pause for phase-by-phase confirmation. Ask again only before expanding its
displayed data, destination, spend, retention, credentials, or production-write
scope.
Use seeded fixtures and resettable sandboxes when the decision requires
repeatability. Treat production writes as a separate envelope from model and
training writes. Follow the repo public boundary in
../../docs/privacy-and-data-boundaries.md
for prompts, completions, tool outputs, traces, datasets, repo paths, and
secrets. Do not print or commit sk_* values; let
../use-understudy-gateway/SKILL.md
inject them into the child process only.
-
Confirm it is agentic and pick the lens. Inspect the workload for a
multi-turn loop and tool calls. State the fixed tool set and policy model for
route comparisons, plus any separately isolated implementation variable.
Then classify: do any tools mutate state? Read-only →
references/read-only-search.md;
state-mutating →
references/state-mutating-workflows.md.
If it is single-output, hand back to capture-evidence.
-
Adopt a runnable harness. Read-only loops use a verifiers environment
(vf.Environment + Rubric; the vf-eval command is one supported
harness) or another replayable provider-native/application runner.
State-mutating workflows use an existing benchmark/sandbox runner with a
deterministic reset (seeded state, fixed API schemas, fixed policy docs,
final-state validator). A small deployable app or harness change is allowed
when it directly addresses a measured failure; version it and evaluate it as
a distinct arm instead of silently changing the comparison. Either way,
capture the runnable contract into the
.understudy/capture-evidence/ artifact contract — each reference documents
the env → artifact bridge for its shape.
Before using model scores, pass the shared evaluation evidence gates in
../capture-evidence/references/evaluation-evidence-gates.md.
In particular, run a synthetic read-then-write trajectory through the exact
driver for every model family. The driver must append the read result,
continue the loop, execute the terminal write, and score final state; an
intermediate tool call is never a no-op verdict.
If Desktop produced understudy.environment_proposal.v1 from a JSONL drop,
treat status: executable as meaningful only after
understudy training validate-environment-proposal --proposal <path> passes.
An Understudy draft with status: needs_verifier is a proposal, not a
harness or score; author the missing parser/environment/oracle/sentinels and
rerun deterministic validation before any model comparison.
-
Define multi-objective success. Quality is a per-criterion LLM-judge or
final-state rubric that returns natural-language why/what-to-change
feedback, not a bare score. Latency and cost come from the rollout records
(turn counts, tool/API call counts, timing, token usage) — read those
instead of inventing a meter. State-mutating workflows add a
side-effect-safety axis (forbidden writes, invalid requests, retries).
Record the axes, workload-specific acceptable-regression/non-inferiority
bands, and any hard constraints in metric.json. Contract requirements and
safety requirements marked hard by the developer remain zero-tolerance;
statistical or multi-objective tradeoffs cannot waive them.
-
Freeze determinism. Read-only: live tool calls are non-deterministic, so
freeze the query set and snapshot/cache the tool outputs so the harness
replays reproducibly and the holdout stays clean. State-mutating: record a
deterministic reset — seeded state, schemas, policy docs, task rows, allowed
endpoints, clock, seed, network boundary — so each run starts from a known
state and emits a request log.
-
Run the incumbent baseline before optimizing. Execute the frozen harness
on the sanctioned train/dev set, write per-task results, and bind
harness_sha256, metric_sha256, and splits_sha256 into baseline.json.
Confirm coverage across completed-execution strata, including rare or
high-consequence tool paths, and inspect the actual trajectories behind at
least one pass, each reported failure class, each surprising delta, and a
counterexample to the proposed headline. Do not optimize until the baseline
is measured and those checks pass. A smaller pilot may validate plumbing, but
it is not the baseline evidence. Expand the cohort whenever estimates remain
unstable, important strata are underfilled, or review discovers new failure
classes.
-
Attribute the multi-turn gap before intervening. Read the rollouts, not
just the final score, and tag where reward is lost: wrong tool/endpoint,
wrong argument value, result-propagation (mis-copying a value a prior
tool returned into a later call), failure to recover from a tool error,
forbidden or missing writes, or non-termination. Single-turn /
next-tool-call imitation scores are a leading indicator only — they cannot
see result-propagation, recovery, or termination, which exist only inside
the running environment. Let the attribution pick the highest-leverage rung
likely to close the gap: output-contract repair (prefill / format / parser /
schema), tool-access or endpoint-catalog repair, model A/B, prompt / GEPA
(automatic prompt evolution), supervised fine-tuning/distillation, or RL.
Rank the eligible rungs by expected objective gain, confidence, time, spend,
and reversibility; start with the highest expected value rather than a fixed
sequence. Use a cheaper rung first only when evidence says it can solve the
attributed failure.
-
Candidate intervention — compare models and routes. When attribution
points to model capability or provider/runtime behavior, compare deployable
candidates through any backend that can honor the frozen contract. The
Understudy CLI is one route:
understudy models list --json
understudy workloads route <workload-id> --project-id <project-id> \
--model-id glm-5.1 --traffic-pct 100
understudy run -- <harness command>
List public model options, route the project workload to a chosen model,
then run the frozen harness through the gateway with understudy run; a
provider-native or existing application runner is equally valid when it
preserves the same rows, prompt, tools, metric, seed/reset, and data
boundary.
Compare quality vs latency vs cost (vs side-effect safety) across candidates
under the decision contract and pick the route you would ship. For keyless
accounts, prefer a managed-catalog sweep on a cleared/no-route workload
before traffic-split A/B. Prerequisite for a traffic split: the non-routed
passthrough share needs a configured managed provider credential or BYO key
so untouched traffic still completes. Clear a route with --clear. Routing
detail lives in
../use-understudy-gateway/SKILL.md.
For state-mutating workflows, A/B is often simpler: run the same harness
rows twice with only the model changed (see the reference).
-
Candidate intervention — repair prompt, tools, contract, or harness. When
the attributed gap is instructional or structural, test the smallest
deployable correction: prompt/GEPA, output parser or schema, tool
descriptions/retrieval, retry policy, or a versioned app/harness change.
If a cheaper model wins on latency and cost but trails on quality, a
train/dev-only GEPA pass against the feedback-rich rubric may close the gap
while keeping the latency/cost win. When the workload already lives in a
promoted benchmark dir with frozen splits, run the automatic loop directly:
understudy runs execute --benchmark <benchmark-dir> --watch
understudy benchmarks evolve <benchmark-dir> --model <candidate-id> --budget-runs 6
evolve is GEPA-style prompt evolution over prompt_overrides run arms:
an authoring model proposes system-prompt suffixes from the failure
evidence (per-class tool-call rejection counts and unmet contract
obligations read from the run journals), each proposal runs as a labeled
override arm, and every generation is recorded in evolution.jsonl
(suffix + sha256 + scores). See
../../docs/prompt-evolution.md for the
loop and budget guidance. Claim rules mirror
../optimize-workload/SKILL.md: it evolves
on train, selects the champion on dev, touches the sealed holdout exactly
once for the final champion-vs-bare run, and reports quality improvement only
when that holdout run's paired 95% CI clears the prespecified superiority
threshold. A route may still be selected after demonstrated quality
non-inferiority or by a prespecified multi-objective rule, but a no_win/
unverified/inconclusive superiority verdict must never be presented as an
improvement. For single-output workloads, hand off to
../optimize-workload/SKILL.md instead;
never tune on holdout.
Prompt experiments as run arms (prompt_overrides). When the workload
lives in a trace-compiled benchmark directory, test a prompt candidate
without touching the app: queue a run whose request carries
prompt_overrides — each {arm_label, model, system_prompt_suffix} arm
appends the suffix file's text to every task's system prompt at rollout
time (understudy runs queue --prompt-override <arm_label>=<model>=<suffix-file>). The canonical use is the SOP-suffix
experiment: distill the workload's standard operating procedure (order of
operations, argument conventions, termination rules) into a short suffix
and run it as its own arm beside the bare model on the same frozen tasks —
one run, same scorer, per-arm rows. Operating detail in
../operate-benchmark-lab/SKILL.md.
-
Candidate intervention — supervised fine-tuning or distillation. Select
this early when the failure evidence supplies learnable correction pairs,
teacher trajectories, or deterministic verifier labels and a trained
candidate can be evaluated in the same end-to-end harness. Do not require
model A/B or prompt optimization to fail first. Freeze the dataset lineage,
keep holdout sealed, declare provider/data/retention/spend bounds, and compare
the trained route against the incumbent under the same hard safety and
contract constraints. Treat next-action imitation as diagnostic unless the
end-to-end rollout also validates result propagation, recovery, termination,
and final state.
-
Escalate to RL only as a true handoff, behind three gates. When the
residual is genuinely stateful multi-step behavior, route to
../prepare-verifier-handoff/SKILL.md.
First confirm: (a) the attribution in step 6 shows cross-turn reasoning
is the residual, not format/argument-value (which supervised or deterministic
repairs can address); (b)
the reward is dense, not strict — a binary/strict reward can be constant
within a group, giving zero advantage and no gradient (paid-for, wasted
steps); and (c) the model has a first-class multi-turn GRPO trainer and
renderer (e.g. NVIDIA Nemotron-3 does; Google Gemma-4 does not yet), or
the RL run is wasted before it starts. This repo never runs that training.
Capture evidence before you optimize, exactly as the rest of the MVP loop
requires (see ../understudy/SKILL.md). The decision
must rest on a measured baseline, and any savings statement needs the
claim.json packet that optimize-workload enforces.
