running-experiment-matrices

name	running-experiment-matrices
description	Use when running, extending, or interpreting quality (E*), grammar (X*), perf (P/Q/R), phase, scaling, or mixture experiment matrices

Running experiment matrices

Overview

Matrices are the intended way to land levers: isolatable IDs, shared gate policy, JSON + markdown scoreboards. Implement a lever, run the matching matrix subset, then document results.

REQUIRED AFTERWARD: documenting-experiment-results. REQUIRED FOR SHIP CLAIMS: honest-ship-eval.

Matrix index

Matrix	Script	Spec	Results JSON
Quality E0–E75 (Vn sets)	python -m scripts.run_quality_matrix	docs/design/quality-experiment-matrix.md	docs/design/quality-matrix-results.json
Grammar X0–X8	python -m scripts.run_grammar_matrix	same (X section)	docs/design/grammar-matrix-results.json
Perf P/Q/R/PG	python -m scripts.run_perf_matrix	docs/design/perf-experiment-matrix.md	docs/design/perf-matrix-results.json
Phase A/B/C	python -m scripts.run_phase_pipeline	quality matrix notes	docs/design/phase-abc-results.json
Baseline seeds	python -m scripts.reproduce_baseline	quality matrix	docs/design/baseline-reproduction-results.json

Design contracts for levers live beside research docs (research-lineage.md, speculative-denoising.md, dsl-native-tokenizer.md, …).

How to run (typical)

# Quality subset (prefer modern matrix set for ship claims)
python -m scripts.run_quality_matrix --matrix v6 --only E53 \
  --steps 80 --device cpu --context-backend scratch --no-design-md-context \
  --rico-limit 32

# Grammar X
python -m scripts.run_grammar_matrix --only X0,X2,X7 --steps 80

# Perf decode matrix (quality guardrails vs P0)
python -m scripts.run_perf_matrix --only P0,Q9,R9,PG --limit 4

Use --only for focused work; full matrices are expensive. Always pass --rico-limit explicitly (CLI default is 32) and record that n in docs; full ship claims need 1500.

Extending a matrix

1. Add the lever in code + config/factory flags.
2. Register a stable experiment ID and run id in the matrix script and the markdown table in the design doc.
3. Run the new ID (plus a baseline control when comparing).
4. Update JSON + measured-results markdown.
5. Link research tags (Implemented / Adapted / …) in research-lineage.md when the lever maps to a paper.

Do not add matrix rows without a runnable script path.

Interpretation rules

• Compare only against runs that share honesty mode, suite sizes, and the same version_stamp component versions (matrix, eval harness, metric, gate policy). After a component bump, find the invalidated rows worth re-running with python -m scripts.verify_version_stamps --stale [--component <id>].
• Perf optimizations fail if parse/fidelity drop >5 points abs vs P0.
• Vacuous guardrails (broken OpenUI bridge zeroing parse) are hard errors — fix the bridge, do not accept empty scoreboards.
• Historical curriculum / gold-leak runs stay labeled invalid for selection.

Completion checklist

• Intended IDs ran (or blockers recorded)
• Results JSON under docs/design/ current and carrying a version_stamp
• Measured-results markdown updated
• Ship/perf pass-fail stated with caveats
• No silent gold channels on honest rows

Parameter efficiency (matrix arms)

• Arms compared to attribute a quality delta must be size-matched (levers.require_size_matched_arms) or must charge the difference. A delta between arms of different capacity is not evidence about the varied factor.
• Model geometry (d_model, n_heads, context_layers, denoiser_layers, the HF backbone) is a declared experiment subject, never a default knob folded into a champion recipe. Capacity ladders are legitimate; a champion that quietly carries a wider geometry is not.
• Report trainable parameters in every matrix row alongside the quality columns, and prefer the smallest arm within the noise band when picking a champion.

Decode invariants (matrix arms)

AGENTS.md § Non-negotiable architecture invariants is goal law; canonical expansion: decode-invariants.md.

• An arm may swap the speculative technique (n-gram ↔ trie ↔ learned ranker) or the prefill schedule; no arm may swap whether output is legal.
• Unconstrained control arms are allowed, must be named as controls, and their rows never feed a ship or promotion decision.
• Arms that set a weakens_constraint lever are diagnostic by construction — label them, and never promote a champion from one.