verify

# Verify — smoke → pass^k → holdout → triggers → budget → triage Current forge state: !`cat .forge/state.json 2>/dev/null || echo NO_STATE` If the line above shows a policy notice instead of JSON (org setting `disableSkillShellExecution`), read `.forge/state.json` with the Read tool before continuing. ## Phase gate (hard) Proceed only when `phase` is `smoke` (start at Step 1) or `verify` (smoke already passed — resume at Step 2). For any other state, run NOTHING and reply with exactly the remediation below, then stop: | Injected state | Reply and stop | |---|---| | NO_STATE | "No forge run in this project. Run `/plugin-forge:forge` to start one." | | `interview` | "Interview incomplete. Run `/plugin-forge:forge`." | | `contract` | "No approved contract yet. Run `/plugin-forge:write-contract`." | | `evals` | "Suite not armed. Run `/plugin-forge:build-evals`, then `/plugin-forge:arm-evals`." | | `armed` | "Nothing built yet. Run `/plugin-forge:build-loop`." | | `building` | "Build loop has not gone green. Run `/plugin-forge:build-loop` to resume it." | | `ship` | "Verification is already green. Run `/plugin-forge:ship`." | Read `plugin_name` and `plugin_dir` from the injected state and use them in every command below. If the state file looks corrupted or stale, run `forge-eval doctor` and re-check before deciding. ## Ground rules - **The suite is the contract.** Never edit anything under `evals/` or `evals-holdout/` during verification. Propose needed suite changes in `.forge/eval-change-requests.md`; only `/plugin-forge:arm-evals` executes them (version bump + re-freeze). - **Plugin edits are allowed** — that is what verification feedback is for — but they invalidate earlier green steps: after a description edit, re-run Step 4 (and re-check Step 5); after any behavior edit, re-run Steps 2–6. - **Run suites in background Bash.** A real suite exceeds foreground Bash timeouts. Launch every `forge-eval run` with `run_in_background: true`; when the job completes, read its exit code and the final `FORGE_EVAL:` stdout line (exit 0 iff `RESULT=PASS`). Progress is resumable from the run's progress file under `runs/<ts>/`. - **Cost cap every run.** Pass `--max-cost-usd <X>` on every invocation, with X from the CONTRACT.md cost sheet (ask the user if the sheet has no verify budget). Report cumulative `cost_usd` per step. - **Infra is not failure.** Trials recorded `timeout` (exit 143) or `unverified` (api_retry) are excluded from pass rates. Re-run affected tasks with `--task <ID>`; never let infra noise flip a gate in either direction. - **Fresh evidence only.** Never reuse build-loop run artifacts as verification evidence; every number reported below comes from runs started by this skill. ## Step 1 — Smoke matrix (entered at phase `smoke`) Catch cheap failures before paying for trials: ``` forge-eval smoke --plugin-dir <plugin_dir> ``` The smoke matrix asserts, in order: 1. **System/init gate** — a `claude -p --bare --plugin-dir <plugin_dir>` session starts with `plugin_errors` and `mcp_server_errors` empty or omitted, and `<plugin_name>` present in `plugins[]`. 2. **Per-skill probes** — every user-invocable skill invoked once via `claude -p --bare "/<plugin_name>:<skill> <fixture args>"`, asserting a non-error result. 3. **Hook fixture pipes** — each recorded event fixture piped into each hook handler, asserting exit code AND output-channel discipline (stderr + exit 2 to block; never exit-2 mixed with JSON output). 4. **Trigger sanity** — 3 should-trigger phrasings per model-invocable skill in fresh sessions, counting activations. Gate: report status **ALL_PASS**. Infra-flagged probes are re-run, not counted. - On a real failure: a plugin defect escaped the build loop. Set phase back to `armed` (snippet in State transitions; regression rule R4 in phase-gates.md), run `/plugin-forge:build-loop`, then restart verification here. - On ALL_PASS: set phase to `verify` and continue. ## Step 2 — Full suite, fresh, at three trials Run capability and regression against the default target: ``` forge-eval run --suite capability --plugin-dir <plugin_dir> --target default --trials 3 --max-cost-usd <X> forge-eval run --suite regression --plugin-dir <plugin_dir> --target default --trials 3 --max-cost-usd <X> ``` Gates: - Both `FORGE_EVAL:` lines end `RESULT=PASS` (exit code 0). - The headline number is capability `pass^3` from the scoreboard line — report it verbatim, alongside the suite version (`version=vN` must match `.forge/freeze.json`). - Regression is expected ≈100%; any regression failure goes to Step 6 triage before proceeding. ## Step 3 — Holdout tranche The holdout tranche lives in `evals-holdout/` and was never seeded into the build worktree — the builder has never seen these tasks or their graders. This is the anti-overfit check (point the harness at the holdout tree with `--evals-dir`, using the suite name from `evals-holdout/suites/`): ``` forge-eval run --suite <suite> --plugin-dir <plugin_dir> --evals-dir evals-holdout --target default --trials 3 --max-cost-usd <X> ``` Gate: `RESULT=PASS`. If the holdout score sits materially below Step 2 (e.g. a pass^3 gap greater than 0.2), treat it as overfitting to visible graders: dispatch triage (Step 6) immediately, expect task hardening via `/plugin-forge:arm-evals`, and re-enter the build loop. The honest claim after this step is "green + holdout + triaged" — never "correct". ## Step 4 — Trigger suite on BOTH targets ``` forge-eval run --suite triggers --plugin-dir <plugin_dir> --target default --max-cost-usd <X> forge-eval run --suite triggers --plugin-dir <plugin_dir> --target crowded --max-cost-usd <X> ``` - `default` = clean `--bare` session: the optimistic ceiling. - `crowded` = distractor plugins loaded via repeated `--plugin-dir`: realistic 1%-listing-budget pressure. This is the honest number. Report precision and recall for BOTH targets — never only default. Gate: crowded-target numbers meet the CONTRACT.md trigger targets; if the contract sets none, require precision 1.0 and recall ≥ 0.8 and surface anything lower to the user. A crowded-only recall miss is a description problem: tweak the component's description keywords (a plugin edit — allowed), then re-run both targets and re-check Step 5, because description edits change always-on cost. ## Step 5 — Token budget (always-on cost) ``` claude plugin details <plugin_name> ``` (If the name does not resolve because the plugin is not installed, run the command from a session started with `claude --plugin-dir <plugin_dir>`.) Compare the always-on token cost — descriptions charged to every session — against the CONTRACT.md cost-sheet budget. Over budget: trim descriptions or demote components (`user-invocable: false` for knowledge, `disable-model-invocation: true` for user-gated actions) per the contract, then re-run Step 4 since descriptions changed. Record both always-on and on-invoke figures in the report. ## Step 6 — Triage dispatch Dispatch the `plugin-forge:triage` agent over the newest `runs/<ts>/` directories from Steps 2–4. Have it sample every failing trial plus 1–2 passing trials per failed task (dual-trajectory inspection). It writes `runs/<ts>/triage.md`, labeling each sampled failure `{agent-error | grader-error | task-ambiguity}` with a recommended fix location. Route by classification: | Finding | Action | |---|---| | `grader-error` | STOP verification. Run `/plugin-forge:arm-evals` (fix the grader, bump the suite to vN+1, re-freeze), then repeat from Step 2 — scores are not comparable across suite versions. | | `task-ambiguity` | Repair the task via `/plugin-forge:build-evals`, re-freeze via `/plugin-forge:arm-evals`, then repeat from Step 2. | | `agent-error` and any gate is RESULT=FAIL | Set phase to `armed` (regression rule R4 in phase-gates.md), run `/plugin-forge:build-loop`, then re-verify from Step 1. | | `agent-error` only, all gates PASS | Tolerated residue within pass^k. Record it in the report and proceed. | ## Ship-ready Declare verification green only when ALL of: 1. Smoke ALL_PASS. 2. Capability + regression fresh at `--trials 3`: RESULT=PASS, pass^3 headline recorded. 3. Holdout RESULT=PASS with no overfit gap. 4. Trigger precision/recall reported for default AND crowded; crowded meets targets. 5. Always-on token cost within the contract budget. 6. Triage report exists with zero unresolved `grader-error` / `task-ambiguity` findings. Then: 1. Write `runs/<latest-ts>/verify-report.md`: the six gates with their numbers, run timestamps, suite version from `.forge/freeze.json`, and total cost. 2. Set phase to `ship` (snippet below). 3. Tell the user verification is green and the next step is `/plugin-forge:ship`. Ship is user-gated (`disable-model-invocation: true`) — never attempt to invoke it; the user runs it. ## State transitions This skill writes `.forge/state.json` exactly twice: `verify` after smoke ALL_PASS, `ship` after all six gates. Regressions to `armed` (defect routing above) use the same snippet. ``` python3 - <new-phase> <<'EOF' import json, sys, datetime, pathlib p = pathlib.Path('.forge/state.json') s = json.loads(p.read_text()) s['phase'] = sys.argv[1] s['updated'] = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds') p.write_text(json.dumps(s, indent=2) + '\n') print('forge phase ->', s['phase']) EOF ```