honest-ship-eval

# Honest ship evaluation ## Overview Ship readiness is **multi-suite** and **honesty-constrained**. Smoke parse alone never proves generalization. Silent gold-placeholder channels invalidate clears. **Policy source:** `docs/design/adversarial-review.md` and `write_ship_gates` / `--ship-gates` in `scripts.evaluate_model`. ## Fixture demo vs ship | Kind | Allowed | Not a ship claim | | --- | --- | --- | | Fixture demo | Tiny upsample, scratch, smoke wiring, CI smoke | Product readiness | | Honest fixture ship path | Matrix clears with inventory-in-prompt, limited `rico_held` n | Production claim | | Production ship | Full `rico_held` (1500 when claimed), HF + DESIGN.md when claimed, full scoreboard + `--ship-gates` | — | Always name which kind you mean in docs and PR text. ## Gate checklist Before asserting pass: 1. Ran `evaluate_model` (or matrix wrap) with **`--ship-gates`** 2. Checked **all** policy suites (smoke, held_out, adversarial, ood, rico_held) 3. Used **placeholder_fidelity** (not soft `placeholder_validity`) for ship bars 4. Confirmed **meaningful parse** (not empty / useless stacks) 5. Confirmed **honest slot contract** — inventory from prompt/DESIGN.md, not hidden `gold.placeholders` when `honest_slot_contract=True` 6. Recorded **suite sizes** (especially `rico_held` n) 7. Cited the run's **version stamp** — `code_commit` plus `gates.ship` / `evals.meaningful_program` versions from `scoreboard.json` / `gates.json` — so the claim stays comparable after the self-improving stack changes (`docs/design/version-stamp-contract.md`) 8. **Updated docs** via `documenting-experiment-results` ## Default honest bars (CLI `--ship-gates`) | Suite | parse | structural | component recall | placeholder_fidelity | reward | | --- | ---: | ---: | ---: | ---: | ---: | | smoke | ≥ 0.66 | ≥ 0.35 | ≥ 0.35 | ≥ 0.25 | ≥ 0.30 | | held_out | ≥ 0.40 | ≥ 0.30 | ≥ 0.30 | ≥ 0.15 | — | | adversarial | ≥ 0.25 | ≥ 0.25 | ≥ 0.20 | — | — | | ood | ≥ 0.25 | ≥ 0.25 | ≥ 0.20 | — | — | | rico_held | ≥ 0.10 | ≥ 0.20 | ≥ 0.15 | — | — | The `parse` column is `meaningful_program_rate` (`ship_gates.py` / `openui_ship_gates_v6.json`); `parse_rate < 1` on a completed suite is an I6 failure (`docs/design/decode-invariants.md`), not a gate. Gate metrics are `meaningful_program_rate`, `structural_similarity`, `component_type_recall`, `ast_beq_rate`, `canonical_beq_rate`, `placeholder_fidelity`, `reward_score`; `contract_recall` is a promotion-lineage metric, not a ship gate. `component_type_recall` is the semantic-density floor (E2): the trivial/empty program scores ~0, so compression/decode changes cannot green gates with shorter-but-emptier output. Do not lower these to green a run. Document a fail and change levers instead. ## Preferred honest recipes ```bash # Focused honest fixture path (V6 champion stack) python -m scripts.run_quality_matrix --matrix v6 --only E53 \ --steps 80 --device cpu --context-backend scratch --no-design-md-context # Explicit ship-gates scoreboard python -m scripts.evaluate_model \ --test-dir outputs/test_data/v1 \ --model twotower --run-id <id> --ship-gates ``` Prefer `--matrix v4+` / V6+ rows (E35/E53 family) for honest claims. Historical V3 template-fill clears that read gold placeholders are invalidated. ## Red flags - Soft smoke-only gates treated as ship - `rico_held` n=23 (or other stub) presented as the 1500 claim - Reward inflated by gold DESIGN.md lint - Curriculum / train seeds isomorphic to smoke (`smoke_align` era) - "Pass" in old `gates.json` without re-eval after remediation ## After eval **REQUIRED:** Follow `documenting-experiment-results` before claiming done. ## Checkpoints (full HF trains) Production HF-context trains must land weights in `hf://buckets/TKendrick/OpenUI/checkpoints/<run_id>/` ([checkpoint-bucket.md](../../../docs/design/checkpoint-bucket.md)). Confirm `train_summary.json` → `checkpoint_bucket` (or an explicit `--no-sync-checkpoints` / scratch rationale). Auth: `HF_TOKEN` / `hf auth login`. Then update [`docs/MODEL_CARD.md`](../../../docs/MODEL_CARD.md) **and** the README “Model card (summary)” with run id, URI, suite metrics, and claim level (demo / scratch matrix / production HF). Ship claims without a card+summary update are incomplete. ## Parameter efficiency (never green a gate by growing the model) - Record **trainable parameters** beside every quality number. A metric without a size is not comparable to one with a different size. - Growing the model to clear a bar is the size-analogue of lowering the bar. Both are forbidden. Document the fail and change levers instead. - A candidate larger than its baseline needs `EG_params` LCB ≥ 1 (`efficiency_gain(..., cost_key="params")`). Unmeasured growth fails closed — wall-time parity is not a size budget. - Readiness claims name the model size they hold at. A number produced by a wider geometry never transfers to the smaller one. ## Timeouts are incompleteness, not quality zeros Decode budget walls (`decode_timeout_seconds`, cooperative deadline, SIGALRM) produce `runtime_timeout` outcomes. They **must not** dilute **any** quality or completed-latency metric — only fields that explicitly track timeouts/runtime incompleteness may include them: | Signal | Denominator / population | | --- | --- | | **All quality metrics** — parse / meaningful / syntax / BEq / fidelity / validity / contract / structural / tree-edit / recall / reward / topology composites / completed latency p50·p95 | **Completed** documents only (`completed_document_n`, `completed_latency_n`) | | `decode_timeout_count` / `decode_timeout_rate` / `incomplete_document_n` / `incomplete_latency_n` / `latency_ms_*_including_incomplete` / `decode_outcome_counts.runtime_timeout` | Runtime incompleteness (budget/interference) | | `empty_prediction_count` | Model abstention on **completed** attempts — **not** timeout stand-ins | | Ship gate `*:decode_timeout_count` | `runtime_failures` — fail closed on any timeout | If every document times out, quality rates and completed latency are **null (unmeasured)**, never fabricated `0.0`. Taxonomy: `decode_outcome.runtime_timeout` (SLM-303). ## Decode invariants (never gate on unconstrained output) `AGENTS.md` § Non-negotiable architecture invariants is goal law; canonical expansion: [decode-invariants.md](../../../docs/design/decode-invariants.md). For evals and gates specifically: - An unconstrained arm (`--unconstrained-control`, HTTP `grammar_constrained=false`, the `current_native` decode-path spec) is a **diagnostic control**. Its output is never certified, never a ship claim, and never a gate input — even when it happens to parse. - Ship-gated configs hold every safe value in `levers.CONSTRAINT_WEAKENING_LEVERS`; `require_constrained_production_config` fails the run before it starts otherwise. Do not route around it. - A gate is never weakened to accommodate a decode that could not stay legal. Fix the decode. - Readiness claims state the ladder rung (I10) they hold at. A rung that is unbuilt is unbuilt — never report it as passed by a lower rung's evidence.