| name | curate-trajectories |
| description | Use to turn loose per-task trajectory JSONs (a Lilac export or local rollout corpus) into a queryable, provenance-tracked, contamination-safe dataset that hard-blocks any selection leaking frozen dev/holdout into a train/RL/distill pool. "curate my trajectories", "which rollouts are train-safe", "exclude the holdout rows before RL", "is this distill pool contaminated", "make a hash-stamped selection", "stop hand-filtering trajectories in bash". |
| metadata | {"understudy":{"mode":"interactive","safety":"local-first","cli_required":false}} |
Curate trajectories
Trajectories pile up as loose per-task JSON across runs (each row roughly
{id, name, score, passed, assertion_results, steps, messages, end_state, finish_reasons, model, toolset, domain, seed, input/output_tokens, cost}). The
moment anyone feeds them to training, distillation, or RL they stop being logs
and become a dataset — and a dataset with no split hygiene silently kills
heldout claims. This worker owns the trajectory dataset as a first-class,
queryable, provenance-tracked, contamination-safe artifact: import → tag splits →
select (hash-stamped) → contamination check → emit a decontaminated pool.
The store is provider-agnostic. The source is Lilac or a local JSON corpus;
this skill owns the hygiene and provenance layer on top, not the browser.
Safety Gates
- Block by default. Any selection destined for a train / RL / distill / SFT
pool is built excluding frozen dev+holdout. Including holdout (or dev) in
such a pool is refused unless the developer passes an explicit override, and
every override is logged in the manifest with who/when/why. Never silently
include. A blocked selection is the safe outcome, not an error to route around.
- The frozen splits are the source of truth. Split membership comes from
splits.json produced by capture-evidence, never re-derived by re-seeding or
re-hashing here. If splits.json is missing or stale, stop and route to
capture-evidence — do not guess membership.
- Local-first, no exfiltration. Index, query, and manifest stay under
.understudy/. Never upload the corpus, never print message bodies, secrets,
or raw payloads — operate on ids, provenance fields, counts, and hashes.
- Reproducible or it didn't happen. Every selection is named and hash-stamped
(selection hash over the sorted row ids + filter expr + corpus hash).
Downstream consumers cite the hash, not a manual grep.
Decision Gate
Use this skill whenever trajectories feed a train / RL / distill / SFT pool, or
whenever you would otherwise hand-filter rows in bash (grep toolset=api, "drop
the seed-7 rows", "keep only passes"). Hand-filtering is the #1 way a heldout
claim silently dies; replace it with a hash-stamped selection. If the trajectories
are only being eyeballed (no downstream training/claim), a manifest is optional —
but the moment a number leaves the building, curate first.
This skill selects on provenance and contamination (split membership, dedup,
outcome conflicts) — it does not judge whether a trajectory is correct or
learnable. That pedagogical filtering (the useful-quadrant / surprise-gap cut)
lives in
local-distillation-lab's pedagogical arm,
which consumes this skill's decontaminated pool.
Flow
- Index + attach provenance. Build a queryable local index (one record per
trajectory) at
.understudy/curate-trajectories/index.jsonl from the run JSONs
(and/or a Lilac export). Carry full provenance: task_id, model, toolset, domain, source_run_id, timestamp, outcome, split (split left unknown until
step 2; back-fill model/toolset/domain from run-level meta when rows
omit them). Record a corpus hash and per-source counts. The task id is the
stable name (real exports put it in name, not the enumeration id). Don't
require a per-row RNG seed — many workloads have none (the initial state is
the seed); contamination safety keys on task_id↔split. Flag rows missing the
fields needed to tag or filter, and keep them out of guarded pools.
- Tag splits from capture-evidence. Load the frozen
.understudy/capture-evidence/splits.json (its splits_sha256 goes in the
manifest). Map each trajectory's task id to train / dev / holdout /
none by the frozen split membership (the rows/row_ids lists). Tag every
index record. Rows in no frozen split are none and are quarantined from
guarded pools unless explicitly admitted.
- Query as a hash-stamped selection. Express the subset as a filter over
provenance fields (e.g.
toolset == "api" and domain == "simple" and outcome == "pass"), evaluated over an allow-listed field set only (no arbitrary code).
Resolve it to a named selection, compute the selection hash, and write a row
manifest — never a loose id list pasted into the next command.
- Contamination check + report. Before emitting, cross-check the selection
against the frozen dev+holdout id sets. Detect: (a) holdout/dev ids inside a
guarded selection; (b) the same task id appearing in both this selection
and the frozen holdout; (c) duplicate trajectories (same content hash); (d)
conflicting outcomes for the same
(task_id, model, seed); (e) rows with
missing provenance. Emit a contamination report; any guarded-pool violation
hard-blocks unless overridden.
- Emit the decontaminated downstream pool. On a clean (or explicitly
overridden) check, write the final selection: a
train-safe / distill-safe
pool that provably excludes frozen dev+holdout, with its selection hash, the
splits_sha256 it was tagged against, the corpus hash, the row count, and the
row id manifest — so the claim survives audit. Hand this to the consumer.
Output Standard
End with:
- index path + corpus hash + per-source/per-split row counts;
- the named selection, its filter expr, its selection hash, and row count;
- the contamination report verdict:
clean, blocked, or override (and if
override, the logged who/when/why and exactly which guarded rows were admitted);
- duplicates, outcome conflicts, and missing-provenance counts found;
- the emitted decontaminated pool path + its hash + the
splits_sha256 it cites;
result_type: curated-selection or blocked;- one recommended next consumer —
compare-trajectories,
local-distillation-lab (including its
pedagogical arm),
prepare-verifier-handoff (its
stage-1 env authoring consumes the train-safe pool), or
recursive-language-model
pedagogical training — with the exact selection hash to pass it.
References
