build-loop

!`cat .forge/state.json 2>/dev/null || echo NO_STATE` # build-loop — launch the goal loop against the frozen suite If the first line of this skill rendered as literal backtick syntax instead of JSON or `NO_STATE` (org policy `disableSkillShellExecution` blocks dynamic injection), read `.forge/state.json` with the Read tool before doing anything else. ## State guard Route on the injected state above. Refuse out-of-order execution and give the exact remediation command. | Injected state | Action | |---|---| | `NO_STATE` | Refuse: no forge project here. Remediation: run `/plugin-forge:forge` | | `phase: interview` | Refuse: pipeline not started. Remediation: run `/plugin-forge:forge` | | `phase: contract` | Refuse: no approved contract. Remediation: run `/plugin-forge:write-contract` | | `phase: evals` | Refuse: the suite exists but is not armed — nothing binds the builder yet. Remediation: run `/plugin-forge:arm-evals` (or `/plugin-forge:build-evals` first if `evals/` is missing) | | `phase: armed` | Proceed — this is the launch phase | | `phase: building` | Refuse to launch a second loop. Check the background forge-build task and report its latest FORGE_EVAL line; if the previous run crashed and state is stuck, run `forge-eval doctor`, then `/plugin-forge:arm-evals` if the freeze needs repair | | `phase: smoke` or `phase: verify` | Refuse: the build already went green. Remediation: run `/plugin-forge:verify` | | `phase: ship` | Refuse: this cycle is done. Remediation: run `/plugin-forge:ship`, or start a new iteration via `/plugin-forge:arm-evals` after suite changes | Read `plugin_name` and `plugin_dir` from `.forge/state.json` for everything below. ## What forge-build does (so the launch is informed, not blind) `bin/forge-build` verifies `phase == armed` and that the `evals/` tree hash still matches `.forge/freeze.json`, then: 1. Creates an isolated git worktree at `.claude/worktrees/forge-build-<ts>`. 2. **Seeds the worktree** with `design/CONTRACT.md`, the frozen `evals/` tree, templates, and — critically — `.forge/state.json` with `phase=building` plus `.forge/freeze.json`. The seeding is what arms the guard hooks inside the child session; without state + freeze in the worktree the "unbypassable rail" silently never engages. `evals-holdout/` is deliberately NOT seeded — the builder must never see the holdout tranche. Within eval trials, the harness additionally stages each task's `reference/` answers outside the trial workspace. 3. Launches a headless builder: `claude -p "/goal ..." --output-format stream-json --verbose` with a scoped `--allowedTools` list (Read, Edit, Write, Glob, Grep, `Bash(forge-eval *)`, `Bash(claude plugin validate *)`, and similar — never a blanket grant). 4. On success cleans up its worktree; on failure leaves it in place, flagged, for triage. (`--keep-worktree` preserves it either way.) While `phase=building`, the PreToolUse guard (`guard-evals.sh`) exit-2 blocks any Edit/Write/Bash mutation under `evals/`, and the Stop-hook sweep (`sweep-evals.sh`) recomputes the tree sha256 against the freeze every turn — these hold even under `--dangerously-skip-permissions`. The same guards apply in the parent project: do not edit `evals/` from this session either. ## Preconditions checklist (walk it, do not skip) 1. **Trust dialog accepted.** The /goal evaluator rides the hooks system; the project must have passed the trust dialog at least once (open `claude` interactively in the project if unsure). 2. **Hooks enabled.** `disableAllHooks` at any settings level, or `allowManagedHooksOnly` in managed settings, makes `/goal` unavailable — fall back to `forge-build --no-goal`. Warn loudly in that case: with hooks disabled the guard hooks and the Stop-hook sweep are dead too, so the freeze is only checked after the fact — instruct verify to recompute the tree hash against `.forge/freeze.json` before trusting any green result, and get explicit user acknowledgment before an unattended run without rails. 3. **MCP OAuth preflight.** For every LIVE server in CONTRACT.md's live-vs-mock table: OAuth cannot happen inside `claude -p`, so authenticate each live server in an interactive session first; tools marked `requiresUserInteraction` are force-denied headlessly (plan mocks or drop those paths); elicitation-using servers need the generated Elicitation auto-answer hook plus `elicitation_answers` fixtures in the affected tasks. Mocked servers need nothing. 4. **Worktree base ref decision.** `worktree.baseRef` is `"fresh"` (branch from the remote default branch) or `"head"` (branch from the current local HEAD). Choose `"head"` when the target project's fixtures or supporting code are uncommitted/unpushed — `"fresh"` would build against a tree that lacks them (forge-build's seeding covers only contract, evals, templates, and forge state). Choose `"fresh"` for a clean default-branch baseline. `baseRef` cannot name a branch; for a pinned branch, create the worktree manually with `git worktree add` and point forge-build at it. 5. **permissions.allow for a long unattended run.** Skill `allowed-tools` grants are per-turn, so a multi-hour run will stall on prompts unless the user pre-approves. Suggest (never silently write) adding to the target project's `.claude/settings.local.json`: ```json {"permissions": {"allow": [ "Bash(forge-eval *)", "Bash(forge-build *)", "Bash(git worktree *)", "Bash(claude plugin validate *)" ]}} ``` Worktrees share approvals saved to the main checkout's `settings.local.json`, so approving once covers the build worktree. The child builder's tools come from forge-build's `--allowedTools`; this list keeps the PARENT (this) session from stalling while tailing and re-running gates. Also run `git worktree list` and `forge-eval doctor` if a previous run crashed — leaked `forge-build-*` worktrees and stuck `phase=building` state must be cleared before relaunching. ## Launch Run forge-build via the Bash tool with `run_in_background: true` — never in the foreground; a real build loop runs 30+ minutes to hours and a foreground call would die at the Bash tool timeout: ``` forge-build --watch ``` While it runs: - Check the background task's output periodically. Surface every `FORGE_EVAL:` line to the user as it appears (each one is an iteration scoreboard) and report cumulative `cost_usd`. - Guard blocks (`BLOCKED by plugin-forge: the eval suite is FROZEN...`) in the stream are the rail working, not an error: the builder tried to "fix" a failing grader. It is instructed to route such desires into `.forge/eval-change-requests.md` instead; if requests accumulate, tell the user a human review + `/plugin-forge:arm-evals` re-arm may be warranted. ## The goal condition (what "done" means, mechanically) forge-build issues a goal of this shape: ``` /goal Build the plugin at <plugin_dir> so that `forge-eval run --suite capability --plugin-dir <plugin_dir>` prints a final FORGE_EVAL line with RESULT=PASS, and `claude plugin validate --strict <plugin_dir>` passes, WITHOUT modifying anything under evals/ (frozen contract, sha256 in .forge/freeze.json). Print the full FORGE_EVAL line each iteration. Stop after 40 turns if not achieved. ``` Three design facts make this work: - **The goal evaluator is TOOL-LESS.** It judges only the conversation — it runs no commands and reads no files. The scoreboard line (shared contract, design doc 5.2) ``` FORGE_EVAL: suite=<name> version=v<N> passed=<X>/<Y> pass^<k>=<0.00> cost_usd=<C> RESULT=PASS|FAIL ``` printed by the harness as its final stdout line is therefore the only admissible proof: it must land in the builder's transcript. A builder that stops printing it stalls the loop — the evaluator can never judge the condition met. - **The invariant clause** ("WITHOUT modifying anything under evals/") is judged by the evaluator AND mechanically enforced by the guard hooks — belt and braces. - **The turn bound** ("stop after 40 turns") is judged from the conversation, so the builder must report progress against it each iteration. ## Suite runs INSIDE the builder session: background Bash + progress file A real suite (tasks x trials) runs far longer than any foreground Bash tool call allows — a foreground `forge-eval run` inside the builder session would time out mid-suite and the iteration would be lost. The builder session must instead: 1. Launch `forge-eval run --suite capability --plugin-dir <plugin_dir>` with Bash `run_in_background: true`. 2. Poll the harness's resumable progress file under the run directory (`runs/<ts>/`) between turns. 3. When the run completes, `cat` the final `FORGE_EVAL:` line from the harness output into the transcript so the tool-less goal evaluator can see the proof. forge-build injects these instructions into the builder prompt. When driving a builder session by hand (debugging, `--no-goal` development), follow the same recipe — it is the difference between an iteration that counts and one that dies at a tool timeout. ## Retry budget - **Soft budget — the turn-bound clause.** "Stop after 40 turns if not achieved" is held by the evaluator from the conversation. Adjust per contract complexity via `forge-build`'s options. - **Hard backstop — the Stop block cap.** `/goal` is a session-scoped prompt Stop hook, and Claude Code overrides any Stop hook after 8 consecutive blocks without progress. Every "not achieved yet" verdict is a block: a builder showing no measurable progress for 8 turns is cut off regardless of the 40-turn clause. In goal mode, forge-build exports `CLAUDE_CODE_STOP_HOOK_BLOCK_CAP=25` into the child goal session's environment to raise that cap for long builds; a fresh `FORGE_EVAL` line with a changing `passed=<X>/<Y>` is the cleanest form of visible progress. - **`--max-iterations N`** bounds the `--no-goal` outer loop. ## StopFailure resilience: snapshot + resume, never respawn API failures do not fire Stop — they fire **StopFailure** (matcher values include `rate_limit`, `overloaded`, `server_error`, ...). A rate-limit spike can therefore kill the loop in a way the goal evaluator never sees. The recipe (shipped as `${CLAUDE_PLUGIN_ROOT}/templates/hooks/stopfailure-resume.sh`): - A StopFailure hook matched on `rate_limit|overloaded` **snapshots resume state** — session_id, worktree path, latest FORGE_EVAL line — into `.forge/` (see the template header for the exact path). StopFailure hook output and exit codes are ignored by Claude Code: the hook records; it cannot block or retry by itself. - **Resume, do not respawn:** `claude -p --resume <session_id>` from the project directory re-enters the build worktree with the active goal restored (turn counters reset). A fresh session instead loses the goal, the iteration context, and the guard-armed worktree association, and re-burns cost rediscovering all of it. - forge-build distinguishes **StopFailure-terminated** runs (resume candidates — the snapshot exists and the last FORGE_EVAL was progressing) from **genuine non-convergence** (triage candidates). Report which one happened; only resume the former. ## Outcomes - **RESULT=PASS** in the final FORGE_EVAL line: forge-build cleans up its worktree. Report the final scoreboard line and total cost, then invoke `/plugin-forge:verify` — it runs the cheap smoke matrix first (phase `smoke`), then the expensive fresh-trial pass^k, holdout tranche, and trigger suite (phase `verify`). The build going green is necessary, not sufficient: the honest claim is "green + holdout + triage", never "correct". - **Non-convergence** (turn bound, block cap, or `--max-iterations` exhausted): the worktree is left flagged for inspection. Dispatch the triage agent over the failing `runs/<ts>/` trials; it labels each failure `agent-error | grader-error | task-ambiguity`. Grader-error findings go through `.forge/eval-change-requests.md` and a human re-run of `/plugin-forge:arm-evals` (version bump + re-freeze) before relaunching `/plugin-forge:build-loop`; agent-errors usually mean more budget or a contract fix. - **StopFailure-terminated:** apply the resume recipe above. - **Crashed/stuck** (`phase=building` with no live process, leaked worktrees): `forge-eval doctor`. Cost honesty: a 20–50-task suite at trials=3 plus a 40-turn builder can run tens of dollars per cycle. `forge-eval --max-cost-usd` hard-aborts runaway suite runs inside the loop; watch `cost_usd` in every FORGE_EVAL line and keep the user informed of the running total.