agentsop-test-fix-loop

# Test-Fix Loop · SOP > One-liner: **The test result IS the next prompt.** Wiring the verifier is > 20% of the work; framing its output as a useful feedback message is 80%. --- ## 1. 何时激活 (Activation Rules) Activate this skill when **any** of the following triggers fire: - The user says "have the agent fix until tests pass", "run lint and tests automatically", "iterate until green", or invokes `aider --auto-test`, `cline --yes`, or an OpenHands-style headless agent. - The task has a **verifiable success command**: a non-zero exit code on failure (pytest, ruff, mypy, eslint, tsc, go test, cargo check, npm run build, make check, …). - You're wrapping a code-editing LLM in a script/CI step and need to decide: *when does the agent return?* - The agent just made an edit and the next message in the loop would be "here's what the verifier said". **Do not activate** when: - Success is **subjective** (writing prose, designing UX). The loop has no feedback signal worth replaying. - The verifier is **slow + interactive** (full E2E suite, multi-min builds). Either async-ify the loop, or run a fast subset (`pytest -x -k changed`) in the loop and gate the slow suite at PR review. - The gate is **human approval**, not a machine check — use the HITL skill. --- ## 2. 核心心智模型 (Core Mental Model) ### 2.1 The test result IS the next prompt The agent's *next turn* is conditioned almost entirely on the message you inject between edit-N and edit-N+1. That message — formatted from `stdout`, `stderr`, `exit_code` — **is the prompt**. The framework labels it "tool result" or "verifier output" but mechanically it is a user-role message the LM consumes verbatim. ⇒ **Framing the feedback dominates the model choice.** A 4000-line raw pytest dump prompts a worse fix than a 30-line "first failing test, traceback, the diff you just applied" digest, *regardless of the model behind it*. ### 2.2 Four primitives ``` +-----------------+ +-----------------+ +-----------------+ +-----------------+ | 1. Verifier | | 2. Capture | | 3. Format | | 4. Iteration | | command | | (stdout + | | feedback | | bound | | | | stderr + | | message | | | | - pytest -x | | exit_code) | | - first error | | - max N tries | | - ruff check | | - timeout cap | | - last K lines | | - escalate / | | - mypy --strict | | - byte cap | | - drop noise | | commit / skip | | - eslint . | | - kill on hang | | - keep colors=0 | | | +-----------------+ +-----------------+ +-----------------+ +-----------------+ ``` Drop any one of these and the loop fails: - No verifier → no signal; the agent guesses "done". - No capture → the model can't read stderr; tracebacks live in stderr. - No formatting → 25k-token output distracts the model (see Aider's 25k context-drift threshold). - No iteration bound → infinite loop; the OpenHands SWE-Bench infinite-loop bug `[oh/6357]` is the canonical failure case. ### 2.3 Why a separate skill (vs "just give the agent a bash tool") Naively: "let the agent run `pytest` and read the output". This breaks because: 1. The agent doesn't know **which** command to run (project-specific). 2. The agent dumps the **full output** into context every iteration, blowing the 25k threshold by iter 3. 3. The agent has **no termination contract** — it'll keep trying after the test passes "to be safe", or keep trying after 30 failures "to be helpful". 4. The agent makes **edits with no audit trail** — if iter 4 was the right fix, you can't bisect because nothing is committed. The loop is a contract: *verifier wiring + output capture + feedback framing + iteration cap + per-fix git commit*. Treat it as one operation, not five. ### 2.4 What "green" means | Verifier returns | Interpretation | Next action | |---|---|---| | `exit 0`, no diagnostics | True success | Commit + exit loop | | `exit 0`, warnings | Soft success | Commit + log; optionally surface to user | | `exit != 0`, parseable error | Actionable failure | Format → feed back → next iter | | `exit != 0`, unparseable (e.g. segfault, OOM) | Environment / infra failure | Escalate; do not re-prompt the LM | | Timeout / hang | Likely infinite loop in code | Kill, format as timeout error, escalate after 1 retry | --- ## 3. SOP 工作流 (Agentic Protocol) ### Step 1 · Wire the verifier command Pick the cheapest verifier that catches the class of bug you care about. Cascade from fastest to slowest: | Stage | Command (concrete) | Catches | Typical latency | |---|---|---|---| | 1. Format | `ruff format --check .` / `prettier --check .` | Style | <1 s | | 2. Lint | `ruff check .` / `eslint .` | Style + obvious bugs | 1–5 s | | 3. Type | `mypy --strict src/` / `tsc --noEmit` | Type errors | 5–30 s | | 4. Test | `pytest -x --ff` / `vitest run --bail 1` | Behavioural | 10 s–min | | 5. Build | `cargo build` / `go build ./...` / `npm run build` | Link / compile | 10 s–min | **Rule**: bind `--lint-cmd` and `--test-cmd` to **stages 1–4 combined into one shell command** (`ruff check . && pytest -x`). This way one feedback message covers all signals; you don't loop separately on lint then on tests. For Aider: ```bash aider --auto-lint --lint-cmd "ruff check ." \ --auto-test --test-cmd "pytest -x --tb=short" ``` For Claude Code / generic agent: ```python result = subprocess.run( ["bash", "-c", "ruff check . && pytest -x --tb=short"], capture_output=True, text=True, timeout=120 ) ``` ### Step 2 · Capture stdout + stderr + exit code (all three) ```python result = subprocess.run( cmd, capture_output=True, text=True, timeout=120, env={**os.environ, "NO_COLOR": "1"} ) captured = { "exit_code": result.returncode, "stdout": result.stdout, "stderr": result.stderr, "timed_out": False, } ``` Common mistakes: - Capturing only stdout — tracebacks in pytest go to **stdout**, but compiler errors in `tsc` / `cargo` go to **stderr**. Always capture both. - Not setting `NO_COLOR=1` — ANSI escapes burn tokens and confuse the model. - No timeout — a single infinite-loop unit test halts the whole agent. - No byte cap — a 50MB `cargo build` log kills your context window. ### Step 3 · Format the feedback message (the load-bearing step) The single biggest lever in this skill. **Don't paste raw output.** Distill to: ``` The verifier failed (exit 1, pytest -x --tb=short). FIRST FAILING TEST: tests/test_auth.py::test_jwt_expiry — AssertionError: expected 401, got 200 TRACEBACK (last frame): File "src/auth.py", line 47, in verify_token if exp < now: return None TypeError: '<' not supported between instances of 'NoneType' and 'datetime' YOUR LAST EDIT touched src/auth.py:40-50. Hypothesis: `exp` is None when the JWT lacks an `exp` claim. Either default it or guard the comparison. ``` Formatting recipe: 1. **First error only.** If there are 12 failing tests, show the first. Subsequent ones often cascade from the first fix. 2. **Last frame of the traceback.** Earlier frames are usually framework noise. 3. **Anchor to the last edit.** "You just changed `src/auth.py:40-50`" makes the model attribute the failure correctly. 4. **Drop unchanged-between-iters noise** — pytest's collection summary, coverage totals, deprecation warnings. 5. **Hard byte cap**: target ≤ 2k tokens of feedback. If a single failure doesn't fit, truncate the traceback middle (keep top + bottom). 6. **No "please fix"** — the framing is enough. Imperative pleas degrade instruction-following in some models. ### Step 4 · Bound the iterations Two limits, both required: - **Hard cap** (`MAX_ITERS = 5` is a sane default; Aider uses ~3, OpenHands uses 50–100 for SWE-Bench). - **Stall detector**: if the **same** error message appears twice in a row, break early — the model is stuck on the wrong hypothesis. ```python seen_errors = [] for i in range(MAX_ITERS): edit = agent.propose_edit(feedback if i else initial_task) apply_edit(edit) git_commit(f"agent: iter {i+1}") # always commit each iter verifier = run_verifier() if verifier["exit_code"] == 0: return Success(iters=i+1) feedback = format_feedback(verifier, last_edit=edit) if feedback in seen_errors[-1:]: # exact repeat return Stall(reason="same error twice", last=feedback) seen_errors.append(feedback) return Escalate(reason=f"exhausted {MAX_ITERS} iters", last=feedback) ``` ### Step 5 · Per-iteration commit (the audit lever) After every edit, before the verifier runs, commit with a structured message: ``` git commit -am "agent[iter 3/5]: tighten exp guard in verify_token" ``` Why mandatory: - If iter 3 made it worse and iter 5 fixed it the "wrong" way, you can bisect with `git log --oneline | head -5`. - The agent never overwrites its own previous attempt — each iter is recoverable. - `git diff HEAD~1` gives the formatter a precise "what you just changed" anchor. Aider's `--auto-commits` (on by default) does this. For non-Aider agents, wrap the loop in commit logic yourself. ### Step 6 · Detect success precisely | Signal | Good or false-positive? | |---|---| | `exit 0` from full verifier command | Good | | `exit 0` but `stderr` contains "warning" | Soft success; surface to user, don't loop | | `exit 0` because no tests collected (`pytest` returns 5) | **False positive** — check `pytest --collect-only` count | | `exit 0` from a `\|\| true`-swallowed command | **False positive** — strip suppression from `--test-cmd` | | `exit 0` but agent disabled / skipped tests to pass | **Critical** — diff for `pytest.skip`, `@pytest.mark.skip`, `xfail` added in last iter | The agent disabling tests to "pass" is the most common pathological success. Add a post-success diff check: `git log -p -1 | grep -E '(skip|xfail|@disable)'`. ### Step 7 · Escalate or commit on exit When the loop exits without success: 1. **Surface the last formatted feedback** — that's the message the human needs to read, not the raw pytest log. 2. **Leave the WIP commits intact** — the user may want to inspect iter 3 even if iter 5 failed. 3. **Tag the escalation reason**: `exhausted`, `stalled`, `env_failure`, `timeout`. The user's fix differs per cause. --- ## 4. 操作模型 (Operation Models) Format: **Trigger → Action → Output → Evidence**. ### OP-1 · Wire a one-shot verifier - **Trigger**: User wants the agent to verify once after editing, no loop yet. - **Action**: Run `<lint> && <type> && <test>` once, capture three-tuple `(stdout, stderr, exit_code)`. - **Output**: Pass/fail signal. If fail, structured digest ready to feed back. - **Evidence**: `[aider/lint-test]` "Aider will try and fix any errors if the command returns a non-zero exit code." ### OP-2 · Format raw verifier output into ≤2k-token feedback - **Trigger**: Verifier failed; about to construct the next prompt. - **Action**: Extract first failure, last traceback frame, anchor to changed file:lines from `git diff HEAD~1 --name-only -U0`. Strip ANSI, coverage, deprecation warnings. Hard byte cap. - **Output**: A digest under 2k tokens with a hypothesis line. - **Evidence**: `[aider/edit-errors]` "Above about 25k tokens of context, most models start to become distracted." Each iteration adds context; keep the per-iter delta tiny. ### OP-3 · Bound the loop - **Trigger**: About to enter or continue a fix loop. - **Action**: Set `MAX_ITERS` (3–5 interactive, 50–100 SWE-Bench), detect stall (same error twice = break), enforce total wall-clock cap. - **Output**: A loop with explicit termination, never `while True`. - **Evidence**: `[oh/6357]` OpenHands infinite-loop bug + `[langgraph/recursion]` "Hitting recursion_limit indicates an underlying design flaw" — same lesson. ### OP-4 · Commit per iteration - **Trigger**: Agent has just applied an edit, before re-running verifier. - **Action**: `git add -A && git commit -m "agent[iter N]: <one-line>"`. Never `--amend`. - **Output**: A bisectable audit trail; iter K is always recoverable. - **Evidence**: `[aider/git]` per-edit auto-commit; `[cline/auto-approve]` Cline mirrors the same "edit→commit→test" rhythm. ### OP-5 · Detect success without false positives - **Trigger**: Verifier exits 0. - **Action**: Confirm (a) tests were actually collected (`pytest` exit 5 ≠ success), (b) no test was newly skipped/xfailed in the last commit, (c) no `|| true` suppression in the verifier command itself. - **Output**: Trusted "green" signal. - **Evidence**: pytest exit-code spec; `[aider/lint-test]` formatter wrapper