| name | session-trace-mining |
| description | Mine AI coding-agent session transcripts into structured, gate-validated GPU-kernel optimization records for the wiki. Use when asked to turn vibe-coding sessions, Codex rollout logs, or Claude Code project transcripts into wiki records; to summarise what a kernel-optimization session achieved; to build or extend a session-trace store; or to re-run and validate one. Also use when asked how a session-derived record's number, snippet, or provenance was established. |
Session Trace Mining
Overview
Turns session transcripts into session-trace-1.0 records: one record per change
that measurably moved a metric, answering which operator, what was wrong, what
changed, why it worked at the machine level, the verbatim code, and the gain.
The pipeline splits three ways: scripts do deterministic extraction, an agent
does semantic distillation, and sixteen mechanical gates catch fabrication. The
gates are why the output can be trusted. A model filling fields is the weak link,
so nothing enters the store that cannot be checked against the transcript.
What makes a transcript corpus different from a repository of commits: there is
no repository to check a citation against. The transcript is the corpus. So
provenance is a set name plus a set-relative path plus a per-line digest, and every
number must trace to a span the packet carried — with the agent's own prose
deliberately excluded from that span set.
Why every part of it lives here
The same shape of problem — extract deterministically, distil semantically, gate
mechanically — applies to any corpus of optimisation history. What travels between
such pipelines is design, not code: the three-way split, the declarative
make_schema.py patch list, the packet contract, and the discipline of writing
down every pitfall with the number behind it.
Nothing is imported. This skill is self-contained, deliberately:
| concern | here | why not shared |
|---|
| operator naming, workload families | families.py | the layout gate derives a record's directory from these functions, so a change in another tree would silently refile records |
| milestone selection (the ratchet) | ladder.py | its thresholds decide this store's record set; and an A/B corpus with no ladder pushes back on rules that have no business knowing about it |
| ranking model | score.py | a re-ranked store with no diff to show for it is the worst kind of drift |
Each of the three pins its own behaviour with a self_test() that runs standalone
(python3 families.py, python3 score.py). score.py reproduces the curve
published by the shared ranking model in tools/wiki_score.py for the inputs this
corpus produces — 4% → 0.35, 20% → 0.66, 99% → 1.0, and the 0.35 feedback band —
so scores stay comparable with the committed store by verification rather than by
coupling. If the shared model changes, that self-test is where the divergence
surfaces.
The only paths outside the skill are the two data locations declared in
config.py: where the product goes (kernel_wiki/session_trace/<set>/) and the
dedup / overlap scan over kernel_wiki/records. No logic crosses the boundary.
Setup
One thing has to be configured, because no transcripts ship with this repository:
- point
STM_ROOT at the directory holding your own transcript archive;
- register your sets in
scripts/config.py SETS, replacing the single
example-set placeholder entry. A set is named, never passed as a path, so a
record's provenance survives moving the archive.
Both output locations are derived and need no configuration:
- scratch —
/tmp/session-trace-mining/<set>/: parsed candidates, segments,
packets. Reproducible, never committed. Override with STM_WORKSPACE.
- product —
<gpu-wiki>/kernel_wiki/session_trace/<set>/: records and the
reports that justify them. Override with STM_STORE. It sits beside the
committed store but not inside kernel_wiki/records/: these records use the
derived session-trace-1.0 schema, so filing them into the committed store would
make that store fail its own schema gate. Promoting one means rewriting it
against schema/kernel/schema.json, deliberately and one at a time.
Plain python3 is enough. The schema gate additionally needs jsonschema;
without it that one gate reports SKIP and the other fifteen still run.
Pipeline
S=<skill-dir>/scripts
export STM_ROOT=/path/to/your/transcript-archive
export STM_SET=example-set
python3 $S/make_schema.py --check
python3 $S/ingest.py
python3 $S/recon.py
python3 $S/partition.py
python3 $S/build_packets.py
python3 $S/validate_store.py --verbose
python3 $S/validate_store.py --injection-tests
python3 $S/score_records.py
python3 $S/make_readme.py
Re-run the last two after every distillation batch.
Read reports/recon.md before distilling. It decides what the product can
honestly be: the citable-number share, the diff-coverage share, and how much the
committed store already covers. If a set turns out to have almost no numbers, its
value is mechanism and anti-patterns — do not force a gain claim onto every record.
Two candidate units
The unit is a property of the corpus, not a preference, and it is declared per set
in config.SETS. Getting it wrong yields either nothing or nonsense, so it was
settled by a probe before any schema existed (see references/lessons.md §1–2).
| unit | when | what one record is |
|---|
version-ladder | the run keeps a numbered ladder (memory/vN.json + vN: commit subjects) | one version, assembled across the whole set — one session is one version, so the ladder does not exist inside a single file |
ab-comparison | no ladder | one measured A/B: a variant comparison printed complete in one output, or the same benchmark run either side of an edit |
Evidence tiers
The single most important design decision. Every span carries a tier, and only
three of five may be cited:
| tier | content | citable | caps gain.basis at |
|---|
| T1 | benchmark / profiler stdout | yes | measured |
| T2 | the agent reading back its own notes (cat NOTES.md, git log, Read memory/vN.json) | yes | reported |
| T3 | an agent-authored structured field | yes | reported |
| T4 | agent prose and thinking | no | — |
| T5 | the orchestrator prompt | no | — |
T4 is excluded because admitting it makes the fabrication gate vacuous: the
agent's invented number becomes its own proof. T5 is excluded because those
prompts state the target percentage, which would license any number near it.
The sixteen gates
schema · ids · layout · provenance · verbatim · no-fabrication ·
direction · raw-isolation · relations · index · evidence-tier ·
diff-coverage · unit-normalization · wiki-overlap · pairing-integrity ·
anonymization
The six that carry the weight:
- provenance — re-resolves the set by name, the file by set-relative path, and
every cited line by
sha256(raw line)[:12]. Retargeting a citation fails; moving
the whole archive to another absolute path still passes. A cited line with no
digest is a failure too — without that clause the check silently does nothing,
which is what the line-shift injection caught.
- verbatim —
implementation.snippet must appear literally in
packets/<seg>.diff. The gate and the distiller read the same file on purpose.
- no-fabrication — every number in
worth.gain must be in the packet's
evidence_text, or derivable from it by one of exactly two closed-form rules
(before/after, or speedup=Nx). Derivation from arbitrary pairs of pool
numbers is deliberately not allowed.
- unit-normalization — the delta must be reproducible from the measured levels,
and no absolute time may appear inside
worth.gain.
- wiki-overlap — nothing the committed store already covers: a colliding
(operator, version) dedup key, a record id, or an episode_key. It scans
kernel_wiki/records, and an empty scan is a failure, not a pass — an
overlap gate whose index resolved to nothing would print OK forever.
- anonymization — the served layers must name no person, host or corpus path.
Session transcripts are full of
/home/<user>/..., and payload is what gets
served. The record id is checked separately, because a home path flattened into
a slug has no slash left for the path pattern to catch.
Never weaken a gate to make records pass, and never let a distilling agent edit
scripts/. When a gate looks wrong, verify by injection:
validate_store.py --injection-tests mutates a record (and, where the error lives
there, its packet) and asserts that the named gate complains. Adding a gate without
an injection test is how a store ends up falsely green — two of the eleven cases
here were asleep on their first run.
Distillation
Spawn agents with references/distill-brief.md verbatim, substituting the
placeholders. Batch by set and record type so a failure has a small blast radius,
and point every agent at the one record that already passes as the worked example.
Require each agent to run validate_store.py itself and iterate to green, and to
report which fields the packet was too thin to fill and which gate blocked
it. That report is the main signal for improving the pipeline; treat a batch that
reports no difficulties with suspicion.
When several agents write into one store concurrently, tell them explicitly to
ignore gate failures naming records they do not own.
Porting to another corpus
Everything is corpus-agnostic except two places:
scripts/config.py SETS — register the set: its path under the archive
root, its transcript format, its candidate unit, and default scope. Defaults are
fallbacks only; ingest.py detects hardware and DSL and records which happened
in arch_basis / dsl_basis.scripts/transcripts.py — the only file that knows how a session log is
shaped. A third agent product means one new parse_* function returning the same
Event stream, plus a branch in detect_format. Everything downstream sees
events and never a raw line.
Two corpus-specific details that will need attention on a new corpus: how a
long-running benchmark's output is linked back to the command that launched it (in
Codex logs it is a SESSION_ID=N handshake), and which label words name a whole
implementation rather than a knob (partition.IMPL_LABEL_RE).
Decide the unit before writing any schema. It is what ids, pairing,
dedup_key and the whole worth.gain ladder key on, so getting it wrong means
re-doing the schema, the packets and three gates. Three probes exist for exactly
that decision and should be re-run on a new corpus:
python3 $S/probe_versions.py <transcript> [...]
python3 $S/probe_set.py <set-root>
python3 $S/probe_ab.py <set-root>
probe_versions.py answers "can I see versions in one file"; probe_set.py
answers the question that actually matters for a ladder, since one session is one
version; probe_ab.py sizes the fallback. Write the pass bars down before running
them, and if a corpus fails its bar, change the unit rather than the bar.
Resources
references/lessons.md — read before starting. Every pitfall found while
building this, with the measured numbers behind each: why one session is one
version, why the codex sets cannot be paired before/after, the s-for-seconds
trap, the table column that inherited a unit it did not have, and the two gates
that were asleep.references/distill-brief.md — the agent brief template.assets/schema/session-trace-1.0.schema.json — the record schema, generated by
make_schema.py from clean-1.3.frozen.json, which is a pinned byte copy of
this repository's schema/kernel/schema.json.
