| name | compare-trajectories |
| description | Use when you need to know HOW two model runs differ behaviorally on the same tasks, not just THAT one scores higher — per-task trajectory diffing that classifies the gap as persistence/recovery, knowledge, or format/parsing. "why does the bigger model pass these", "is this gap RL-shaped", "diff these two trajectory runs", "where do the trajectories diverge", "what would distillation buy me". The behavioral complement to compare-model-sweep. |
| metadata | {"understudy":{"mode":"interactive","safety":"local-first","cli_required":false}} |
Compare Trajectories
Use this worker when
../compare-model-sweep/SKILL.md has already
told you that model A and model B score differently on a frozen eval, and the
next question — the one that decides whether to spend RL budget — is HOW their
trajectories differ, step by step, on the same tasks. A pass-rate delta is a
scalar; it cannot tell you whether the stronger model wins because it tries
longer, because it knew the right endpoint, or because the weaker model's
tool-calls were malformed. Those three answers point at three different fixes —
RL, retrieval/world-knowledge, or decoding/prompt — and only a per-task
trajectory diff separates them.
This skill aligns two (or more) run exports by task id, builds the outcome-delta
matrix, computes behavioral divergence on shared tasks, classifies each
reachable-gap task, and counts how many clean warm-start trajectories the
comparison yields. It reads trajectory JSON that already exists on disk; it does
not run models. To produce the trajectories, run
../run-local-model-lab/SKILL.md or
../compare-model-sweep/SKILL.md first.
Safety Gates
- Local-first, no upload. Trajectory exports contain full message bodies and
tool I/O. Keep the diff, the matrix, and the report under
.understudy/; do not
paste trajectories or task content into external services.
- Flag small N. A reachable-gap set of 3 tasks is an anecdote, not a
capability claim. State the shared-task count and the per-class count on every
verdict, and refuse to generalize below a stated floor (default N<10 →
"directional only").
- Segregate frozen holdout. If either export includes holdout/frozen rows,
say so and split the matrix into train/dev vs holdout sections — never
warm-start from holdout trajectories, and never report a holdout-derived gap as
a training signal. Prefer a hash-stamped selection from
../curate-trajectories/SKILL.md.
- No vibes. "B is better" must be backed by the per-task matrix and the
divergence metrics, never by reading a few transcripts. The gap classification
is a hypothesis to confirm, not proof — label it as such.
- Make all token/cost/step claims from the export fields, not memory.
Decision Gate
Use this skill when you need to know WHY two runs differ. If you only need the
scalar frontier (pass-rate, latency, cost, reliability) to pick a route, use
../compare-model-sweep/SKILL.md instead —
this skill is its behavioral complement, not a replacement. Both runs must come
from the same frozen task set, harness, and tool-access mode; if they don't,
stop — a trajectory diff across different tasks is meaningless.
This skill diagnoses the RL-vs-not gap behaviorally (trajectory divergence,
recovery events). The learnability cut on the resulting warm-start set —
surprise-gap / concentration measured under the student — is a different
instrument, owned by
../local-distillation-lab/SKILL.md and
its pedagogical arm (references/pedagogical-arm.md), not here. They compose:
behavioral verdict here, learnability cut there.
Flow
-
Load two runs. Take two run JSON exports (A = baseline/weaker, B =
candidate/stronger by convention). Each has per-task records with
{id, name, score, passed, assertion_results, steps, messages (role + tool_calls), end_state, finish_reasons, input/output_tokens, cost}. Validate
both share a schema and a task-id space; record the export paths, model ids,
and harness/tool-access mode. More than two runs → run pairwise vs the chosen
baseline.
-
Align by task id. Inner-join on the stable task name (real exports put
it in name; id is only an enumeration index). Report the shared-task count,
any ids
present in only one run (and why — crash, timeout, missing), and which ids
carry a holdout/frozen flag. Drop unmatched ids from divergence math; list them.
-
Build the outcome-delta matrix. Cross-tabulate passed: A✓B✓ (both pass),
A✗B✗ (both fail), A✗B✓ (the reachable-gap set — what B unlocks), and A✓B✗
(regressions — what B loses). Emit agreement count, disagreement count, and the
two named id lists. The reachable-gap set is the input to steps 5–6.
-
Behavioral divergence on shared tasks. Per shared task, compute from the
trajectories: steps-to-done (both); finish_reasons distribution bucketed to
{completed, max_steps, gave_up, error}; distinct tool calls / endpoints
attempted; recovery-after-error events (did it retry after a 404 / empty result
/ error, or quit the next step?); and first-divergence step — the first
step index where the two trajectories' tool-call sequences stop matching.
Summarize as distributions, not raw transcripts.
-
Classify the gap. For each reachable-gap task (A✗B✓), label the gap as:
- persistence/recovery — B reached the same tools/endpoints A did but A
stopped early (gave_up / fewer steps / no retry after error) while B retried
and recovered. RL-learnable — the policy can be trained to try longer.
- knowledge — B called a correct endpoint/tool A never attempted at all.
Not RL-addable — RL won't inject world knowledge; needs retrieval, a tool
hint, or a stronger base.
- format/parsing — A's failure is malformed tool-calls / schema-invalid
args / unparseable output, not a strategy gap. Fixable by prompt/decoding,
not RL.
Emit the per-class breakdown and per-task evidence (which signal fired). Treat
the label as a hypothesis; spot-check a redacted sample per class to confirm.
-
Warm-start yield. Count the A✗B✓ tasks where B's passing trajectory is
clean (passed, no error-recovery thrash beyond a threshold, valid
tool-calls, terminated completed). These are high-quality distillation /
warm-start examples — feed them to
../local-distillation-lab/SKILL.md.
Exclude any holdout-flagged tasks from the yield count.
-
Report. Write the matrix, divergence summary, gap-class breakdown,
warm-start count, and the RL-shaped verdict to
.understudy/trajectory-diffs/<timestamp>/.
Repeat-replay stability
A single run per model can mistake sampling noise for a behavioral gap. Before
classifying gaps — and always before a candidate graduates toward live traffic
— re-run the candidate on the same frozen rows in incremental batches. Three
repeats are a plumbing smoke, not a universal N; size the replay for the agreed
instability tolerance and expand near the threshold. Then classify: all repeats
match → stable; some → borderline; none → stochastic (exclude from gap
classification and the warm-start yield). Above ~5% unstable, fix decoding
(temperature, seed) or downgrade verdicts to directional-only. The dispositions
feed the pre-ramp gate in ../ramp-and-verify/SKILL.md
(stable → serve, borderline → shadow, stochastic → incumbent fallback).
Use the data-sufficiency stopping rule in
../capture-evidence/references/evaluation-evidence-gates.md.
Token-logprob lens
When two same-family runs (e.g. quantized vs full-precision) diverge on the
same row and you need the token-level why — which token the weaker model got
wrong and how confident it was — see
references/logprob-lens.md. It documents the
renderer in scripts/render-logprob-compare.mjs
and the private logprob-sidecar artifacts it requires; nothing public in this
repo produces those artifacts yet, so treat it as an optional lens, not a step.
Output Standard
End with the diff path and: the two run ids + harness/tool-access mode;
shared-task count and N caveat; the outcome matrix (A✓B✓ / A✗B✗ / A✗B✓
reachable-gap / A✓B✗ regressions) with named id lists; the behavioral
divergence summary (steps-to-done, finish-reason buckets, recovery events, median
first-divergence step); the gap-class breakdown (persistence/recovery vs
knowledge vs format/parsing over the reachable-gap set); the warm-start yield
(clean B-passes of A-failures); and an explicit "is this gap RL-shaped?"
verdict — RL-shaped when persistence/recovery dominates, not RL-shaped when
knowledge or format dominates — stated as a hypothesis with the N caveat and any
holdout segregation. If regressions (A✓B✗) exist, surface them; a stronger model
that loses tasks is not strictly better.
References
