honest-ship-eval

name	honest-ship-eval
description	Use when evaluating models, writing or interpreting ship gates, claiming readiness, changing parse/fidelity/reward metrics, or deciding fixture-demo vs production ship

Honest ship evaluation

Overview

Ship readiness is multi-suite and honesty-constrained. Smoke parse alone never proves generalization. Silent gold-placeholder channels invalidate clears.

Policy source: docs/design/adversarial-review.md and write_ship_gates / --ship-gates in scripts.evaluate_model.

Fixture demo vs ship

Kind	Allowed	Not a ship claim
Fixture demo	Tiny upsample, scratch, smoke wiring, CI smoke	Product readiness
Honest fixture ship path	Matrix clears with inventory-in-prompt, limited rico_held n	Production claim
Production ship	Full rico_held (1500 when claimed), HF + DESIGN.md when claimed, full scoreboard + --ship-gates	—

Always name which kind you mean in docs and PR text.

Gate checklist

Before asserting pass:

1. Ran evaluate_model (or matrix wrap) with --ship-gates
2. Checked all policy suites (smoke, held_out, adversarial, ood, rico_held)
3. Used placeholder_fidelity (not soft placeholder_validity) for ship bars
4. Confirmed meaningful parse (not empty / useless stacks)
5. Confirmed honest slot contract — inventory from prompt/DESIGN.md, not hidden gold.placeholders when honest_slot_contract=True
6. Recorded suite sizes (especially rico_held n)
7. Cited the run's version stamp — code_commit plus gates.ship / evals.meaningful_program versions from scoreboard.json / gates.json — so the claim stays comparable after the self-improving stack changes (docs/design/version-stamp-contract.md)
8. Updated docs via documenting-experiment-results

Default honest bars (CLI --ship-gates)

Suite	parse	structural	component recall	placeholder_fidelity	reward
smoke	≥ 0.66	≥ 0.35	≥ 0.35	≥ 0.25	≥ 0.30
held_out	≥ 0.40	≥ 0.30	≥ 0.30	≥ 0.15	—
adversarial	≥ 0.25	≥ 0.25	≥ 0.20	—	—
ood	≥ 0.25	≥ 0.25	≥ 0.20	—	—
rico_held	≥ 0.10	≥ 0.20	≥ 0.15	—	—

component_type_recall is the semantic-density floor (E2): the trivial/empty program scores ~0, so compression/decode changes cannot green gates with shorter-but-emptier output. Do not lower these to green a run. Document a fail and change levers instead.

Preferred honest recipes

# Focused honest fixture path (V6 champion stack)
python -m scripts.run_quality_matrix --matrix v6 --only E53 \
  --steps 80 --device cpu --context-backend scratch --no-design-md-context

# Explicit ship-gates scoreboard
python -m scripts.evaluate_model \
  --test-dir outputs/test_data/v1 \
  --model twotower --run-id <id> --ship-gates

Prefer --matrix v4+ / V6+ rows (E35/E53 family) for honest claims. Historical V3 template-fill clears that read gold placeholders are invalidated.

Red flags

• Soft smoke-only gates treated as ship
• rico_held n=23 (or other stub) presented as the 1500 claim
• Reward inflated by gold DESIGN.md lint
• Curriculum / train seeds isomorphic to smoke (smoke_align era)
• "Pass" in old gates.json without re-eval after remediation

After eval

REQUIRED: Follow documenting-experiment-results before claiming done.

Checkpoints (full HF trains)

Production HF-context trains must land weights in hf://buckets/TKendrick/OpenUI/checkpoints/<run_id>/ (checkpoint-bucket.md). Confirm train_summary.json → checkpoint_bucket (or an explicit --no-sync-checkpoints / scratch rationale). Auth: HF_TOKEN / hf auth login.

Then update docs/MODEL_CARD.md and the README “Model card (summary)” with run id, URI, suite metrics, and claim level (demo / scratch matrix / production HF). Ship claims without a card+summary update are incomplete.

Parameter efficiency (never green a gate by growing the model)

• Record trainable parameters beside every quality number. A metric without a size is not comparable to one with a different size.
• Growing the model to clear a bar is the size-analogue of lowering the bar. Both are forbidden. Document the fail and change levers instead.
• A candidate larger than its baseline needs EG_params LCB ≥ 1 (efficiency_gain(..., cost_key="params")). Unmeasured growth fails closed — wall-time parity is not a size budget.
• Readiness claims name the model size they hold at. A number produced by a wider geometry never transfers to the smaller one.

Timeouts are incompleteness, not quality zeros

Decode budget walls (decode_timeout_seconds, cooperative deadline, SIGALRM) produce runtime_timeout outcomes. They must not dilute any quality or completed-latency metric — only fields that explicitly track timeouts/runtime incompleteness may include them:

Signal	Denominator / population
All quality metrics — parse / meaningful / syntax / BEq / fidelity / validity / contract / structural / tree-edit / recall / reward / topology composites / completed latency p50·p95	Completed documents only (completed_document_n, completed_latency_n)
decode_timeout_count / decode_timeout_rate / incomplete_document_n / incomplete_latency_n / latency_ms_*_including_incomplete / decode_outcome_counts.runtime_timeout	Runtime incompleteness (budget/interference)
empty_prediction_count	Model abstention on completed attempts — not timeout stand-ins
Ship gate *:decode_timeout_count	runtime_failures — fail closed on any timeout

If every document times out, quality rates and completed latency are null (unmeasured), never fabricated 0.0. Taxonomy: decode_outcome.runtime_timeout (SLM-303).

Decode invariants (never gate on unconstrained output)

AGENTS.md § Non-negotiable architecture invariants is goal law; canonical expansion: decode-invariants.md. For evals and gates specifically:

• An unconstrained arm (--unconstrained-control, HTTP grammar_constrained=false, the current_native decode-path spec) is a diagnostic control. Its output is never certified, never a ship claim, and never a gate input — even when it happens to parse.
• Ship-gated configs hold every safe value in levers.CONSTRAINT_WEAKENING_LEVERS; require_constrained_production_config fails the run before it starts otherwise. Do not route around it.
• A gate is never weakened to accommodate a decode that could not stay legal. Fix the decode.
• Readiness claims state the ladder rung (I10) they hold at. A rung that is unbuilt is unbuilt — never report it as passed by a lower rung's evidence.