tone-model-eval

# tone_model_eval Compares candidate models on a fixed-input copy-pass task and reports which one most closely follows this repo's AGENTS.md → Voice & tone guidance. The eval never changes production guidance or model selection itself — see `out_of_repo_handoff.md` for the separate, out-of-repo steps that act on a positive recommendation. ## Design Every candidate model edits the **identical** "before" text for a given fixture — no model sees a different starting draft than any other candidate for the same fixture. This fixed-input design is load-bearing: it is what makes the per-model scores on a fixture comparable to each other. It does **not** make them comparable to the fixture's original, unknown-provenance author — see the report's "What this eval can and cannot claim" section (`score_outputs.py`'s `SCOPE_BOUNDARY_SECTION`), which every report emits verbatim. Scoring blends two independent axes, recorded separately rather than blended into one number: - **Mechanical** — the existing `style_lint.check_tone_buzzwords` and `check_meta_openers` checks, plus a word-count delta against the "before" text (reusing `doc_quality_policy.check_compression_contract.count_words`). - **Judge rubric** — a fixed 1-5 score on three dimensions (concision, avoids over-explaining, technical fidelity) from an anonymized judge call. The before/candidate text is untrusted, model-produced content, so `build_judge_prompt` wraps both in `<before>`/`<candidate_rewrite>` blocks with an explicit instruction to treat their contents as data to score, not as instructions to follow (prompt-injection resistance). See `judge_rubric.md`. ## Running the eval 1. **Select or extend fixtures.** `fixtures.json` is the fixed comparison-set manifest. Each entry has `id`, `content_type`, `source_path`, `before_commit`, `known_feedback`, and `synthetic` (plus `synthetic_content` when `synthetic` is `true` — a deliberately over-verbose seed draft used when no natural historical example exists for that content type). Validate any change to this file: ```bash python3 .agents/skills/tone_model_eval/score_outputs.py validate-fixtures ``` 2. **Dispatch one copy-pass run per candidate model.** For every fixture, send every candidate model (Fable 5.1, the current default, and any cheaper candidates) the exact same prompt from `copy_pass_prompt.md` with that fixture's "before" text inserted verbatim. Save each model's output to its own file. 3. **Score each output.** For every (fixture, model) pair: ```bash python3 .agents/skills/tone_model_eval/score_outputs.py judge-prompt \ --fixture-id <FIXTURE_ID> --output-file <OUTPUT_FILE> ``` Send the printed prompt to the fixed judge model (or a human judge), save its JSON response to a file, then score the row. `--judge-model-id` is required and explicit (pass the literal `human` for a human judge) so the report can record judge provenance for the bias check in `judge_rubric.md`: ```bash python3 .agents/skills/tone_model_eval/score_outputs.py score \ --fixture-id <FIXTURE_ID> --model-id <MODEL_ID> \ --output-file <OUTPUT_FILE> --judge-response-file <JUDGE_RESPONSE_FILE> \ --judge-model-id <JUDGE_MODEL_ID_OR_human> --rows-file rows.jsonl ``` 4. **Read the report.** Once every fixture/model pair has a row in `rows.jsonl`: ```bash python3 .agents/skills/tone_model_eval/score_outputs.py report \ --rows-file rows.jsonl --fable-model-id <FABLE_MODEL_ID> \ --default-model-id <CURRENT_DEFAULT_MODEL_ID> \ --output-json report.json --output-md report.md ``` `report` first checks that every model in `rows.jsonl` covers the exact same set of fixture ids, with no duplicates, and fails loudly on any missing, unexpected, or duplicate row rather than silently ranking models over a mismatched fixture set. It then applies the pinned adoption thresholds (`score_outputs.py`'s `ADOPT_CONCISION_MARGIN`, `ADOPT_MECH_REDUCTION_PCT`, `CHEAPER_JUDGE_TOLERANCE`, `CHEAPER_MIN_TECHNICAL_FIDELITY`) and states a pass/fail verdict by name for each recommendation arm — including an explicit "no meaningful difference found" verdict when neither adoption threshold is met, which is a valid, reportable outcome, not a blocked eval. ## Scope This skill never opens a content PR and never edits `AGENTS.md`, `style_lint`, `doc_quality_policy`, or any `draft_*` skill — those changes are a follow-up ticket gated on what a run of this eval finds. Its only output is the comparison report (JSON + Markdown) and, when the report recommends acting, a pointer to the separate `out_of_repo_handoff.md` checklist.