| name | running-experiment-matrices |
| description | Use when running, extending, or interpreting quality (E*), grammar (X*), perf (P/Q/R), phase, scaling, or mixture experiment matrices |
Running experiment matrices
Overview
Matrices are the intended way to land levers: isolatable IDs, shared gate
policy, JSON + markdown scoreboards. Implement a lever, run the matching
matrix subset, then document results.
REQUIRED AFTERWARD: documenting-experiment-results.
REQUIRED FOR SHIP CLAIMS: honest-ship-eval.
Matrix index
| Matrix | Script | Spec | Results JSON |
|---|
| Quality E0–E75 (Vn sets) | python -m scripts.run_quality_matrix | docs/design/quality-experiment-matrix.md | docs/design/quality-matrix-results.json |
| Grammar X0–X8 | python -m scripts.run_grammar_matrix | same (X section) | docs/design/grammar-matrix-results.json |
| Perf P/Q/R/PG | python -m scripts.run_perf_matrix | docs/design/perf-experiment-matrix.md | docs/design/perf-matrix-results.json |
| Phase A/B/C | python -m scripts.run_phase_pipeline | quality matrix notes | docs/design/phase-abc-results.json |
| Baseline seeds | python -m scripts.reproduce_baseline | quality matrix | docs/design/baseline-reproduction-results.json |
Design contracts for levers live beside research docs (research-lineage.md,
speculative-denoising.md, dsl-native-tokenizer.md, …).
How to run (typical)
python -m scripts.run_quality_matrix --matrix v6 --only E53 \
--steps 80 --device cpu --context-backend scratch --no-design-md-context \
--rico-limit 32
python -m scripts.run_grammar_matrix --only X0,X2,X7 --steps 80
python -m scripts.run_perf_matrix --only P0,Q9,R9,PG --limit 4
Use --only for focused work; full matrices are expensive. Always pass
--rico-limit explicitly (CLI default is 32) and record that n in docs;
full ship claims need 1500.
Extending a matrix
- Add the lever in code + config/factory flags.
- Register a stable experiment ID and run id in the matrix script and the
markdown table in the design doc.
- Run the new ID (plus a baseline control when comparing).
- Update JSON + measured-results markdown.
- Link research tags (Implemented / Adapted / …) in
research-lineage.md
when the lever maps to a paper.
Do not add matrix rows without a runnable script path.
Interpretation rules
- Compare only against runs that share honesty mode, suite sizes, and the
same
version_stamp component versions (matrix, eval harness, metric,
gate policy). After a component bump, find the invalidated rows worth
re-running with python -m scripts.verify_version_stamps --stale [--component <id>].
- Perf optimizations fail if parse/fidelity drop >5 points abs vs P0.
- Vacuous guardrails (broken OpenUI bridge zeroing parse) are hard errors —
fix the bridge, do not accept empty scoreboards.
- Historical curriculum / gold-leak runs stay labeled invalid for selection.
Completion checklist
Parameter efficiency (matrix arms)
- Arms compared to attribute a quality delta must be size-matched
(
levers.require_size_matched_arms) or must charge the difference. A delta
between arms of different capacity is not evidence about the varied factor.
- Model geometry (
d_model, n_heads, context_layers, denoiser_layers,
the HF backbone) is a declared experiment subject, never a default knob
folded into a champion recipe. Capacity ladders are legitimate; a champion
that quietly carries a wider geometry is not.
- Report trainable parameters in every matrix row alongside the quality
columns, and prefer the smallest arm within the noise band when picking a
champion.
Decode invariants (matrix arms)
AGENTS.md § Non-negotiable architecture invariants is goal law; canonical
expansion: decode-invariants.md.
- An arm may swap the speculative technique (n-gram ↔ trie ↔ learned
ranker) or the prefill schedule; no arm may swap whether output is legal.
- Unconstrained control arms are allowed, must be named as controls, and their
rows never feed a ship or promotion decision.
- Arms that set a
weakens_constraint lever are diagnostic by construction —
label them, and never promote a champion from one.
