| name | test-auto-fix |
| description | Diagnoses failing tests across any project, classifies each failure as a test-bug, prod-bug, or unsure, confidence-gates the fix (auto-apply at >=90%, 80-89 ask, <80 escalate), applies it, and re-runs until green. Surface-driven: reads per-project configuration from a surface file keyed by normalised git remote URL. Bootstrap auto-detects the stack on first run and proposes a surface diff for user approval. Hard-refuses to delete tests, add .skip/.only, or weaken assertions. Regression-detects after every fix: reverts on new failure instead of stacking. Triggers on "fix my failing tests", "tests are red", "auto-fix tests", "heal the tests", "/test-auto-fix".
|
| disable-model-invocation | false |
| argument-hint | [<surface>|<file-path>] [--plan-only] [--max-iterations <n>] |
| license | MIT |
| metadata | {"author":"mthines","version":"1.2.0","workflow_type":"applied","tags":["tests","auto-fix","confidence-gate","regression-detection","guardrails","stack-agnostic","surface-driven","bootstrap"]} |
test-auto-fix
Autonomously diagnose and fix failing tests in any project, with hard
guardrails so the loop never devolves into "make the red go away".
This SKILL.md is the orchestration index.
Load the matching rule file when you need detail — do not preload them.
Always read rules/anti-patterns.md first.
The hard refusals apply to every phase.
Input
The user provides one of:
- Nothing — auto-run every surface defined in the project surface file.
- A surface name (
vitest, unit, integration, etc.) — run only that surface.
- A file path — narrow detection and re-run to that file.
--surface <path> — override the surface file lookup entirely.--plan-only — diagnose and produce the plan artifact; do not write code.--max-iterations N — outer-loop cap (default 5).
The argument is: $ARGUMENTS.
Phase 0 — Resolve the surface
Determine the surface file for the current project. Phase 0 is a hard
two-branch decision; pick exactly one path and follow it through.
Step 1 — Pick the resolution branch
- If
--surface <path> was passed → branch A.
- Otherwise → branch B.
Branch A — --surface <path> override
Use the file at <path> directly as the surface. Do NOT compute the project
key, do NOT scan surfaces/, do NOT run bootstrap.
Still validate per rules/surface-validation.md,
but tolerate the project-key mismatch warning (it's expected when overriding).
When validation passes, jump straight to Phase 1.
Branch B — Auto-resolve from project key
-
Compute the project key per rules/project-keying.md.
-
Look for surfaces/<project-key>.md next to this skill file.
The skill's own directory is resolved by following the symlink chain of
the loaded SKILL.md:
readlink -f "$(dirname "$0")"
If that fails (model-invocation context), default to
~/.agents/skills/test-auto-fix/surfaces/.
-
If no surface file is found → run bootstrap per
rules/bootstrap.md. Bootstrap detects the stack,
proposes a surface diff, waits for user approval, then writes the file.
Do NOT proceed to Phase 1 until the surface exists.
-
If a surface file exists → validate it per
rules/surface-validation.md. If validation
fails, propose an update diff and ask once. If the user declines the update,
escalate — do not run with an invalid surface.
Phase 1 — Detect failures and build the plan
-
Run the detect command from the surface file:
<detect-command>
If --path <file> was supplied, append it to narrow the run.
If the cache-bust-flag is set in the surface and a prior run was cached-green,
append the flag to force a real run.
-
Parse failures using the failure-parser regex from the surface file.
Group by file, then by failure type.
-
If every surface is green: stop immediately. Tell the user. Do not invent work.
-
Write the fix plan to .agent/{branch}/test-auto-fix-plan.md using
templates/plan-artifact.md.
The plan is read-only documentation of intent.
Order: high-confidence test-bug fixes first, prod-bug suspects last.
Phase 2 — Classify each failure (verdict required)
For every failure, apply the rubric in rules/verdicts.md.
Getting this wrong means later phases can corrupt production code to satisfy
a broken test, or "fix" a real regression by patching the test.
Emit exactly one verdict per failure:
test-bug — production code correct; test drifted. Continue to Phase 3.prod-bug — production code regressed; test caught it. Escalate. Do not touch the test.unsure — confidence < 80%. Escalate. Surface the evidence and stop.
Phase 3 — Draft the minimal fix
For each test-bug failure, draft the smallest edit that addresses the
diagnosed root cause.
Hard refusals — full list and detection patterns in rules/anti-patterns.md:
- Deleting any test,
describe, it, or story export.
- Adding
.skip, .only, xit, xdescribe, fit, fdescribe, it.todo, test.todo.
- Replacing strict matchers with loose ones.
- Deleting
expect(...) / assert*() calls.
- Mocking the System Under Test (the module being tested) instead of its deps.
- Wrapping the failing assertion in
try/catch to swallow it.
Phase 3.5 — Confidence gate (before any edit)
Invoke the confidence skill in analysis mode for each proposed fix:
Skill("confidence", "analysis proposed fix: <one-line summary>; verdict: <test-bug>; surface: <surface-name>; risk: <test-only|prod-touch>")
Record the score in the plan artifact.
Full decision matrix: rules/confidence-gate.md.
At a glance:
| Score | Action |
|---|
| ≥ 90 | Auto-apply. Continue to Phase 4. |
| 80–89 | Show the diff, ask once, apply on approval. |
| < 80 | Escalate. Do not write. |
The gate is non-negotiable. Auto mode does not override it.
Phase 4 — Apply + verify (per fix)
-
Apply the edit with Edit — never Bash sed/awk.
-
Re-run only the single failing test using the single-test-command from
the surface file, substituting {file} and {name}:
<single-test-command>
-
If still failing: do not patch over. Revert with:
git restore <file>
Re-classify at Phase 2.
-
If green, and test-provenance-guard is installed: invoke
Skill("test-provenance-guard") on the file. If it reports
tests-by-construction, revert and escalate.
-
Only after both checks pass, mark the failure resolved in the plan.
Phase 5 — test-provenance-guard (optional)
If Skill("test-provenance-guard") is not installed, log:
companion: test-provenance-guard — not available, continuing
Phase 6 — Outer loop
After a batch of fixes:
-
Re-run the full surface using the detect command:
<detect-command>
Regressions hide in untouched files — always re-run the full surface.
-
Compare against the previous failure set per
rules/regression-detection.md:
- Same failures → back to Phase 2 (re-classify; do not repeat same fix).
- Strict subset → continue with remaining failures.
- New failure → revert last fix, re-plan or escalate.
-
Cap at --max-iterations (default 5).
-
On any exit, write a summary section to the plan file.
Phase 7 — Report
Always end with the structured exit summary from
templates/exit-summary.md.
test-auto-fix run
Outcome: <green | escalated | regression-reverted | max-iterations>
Resolved: <N> failures
Escalated: <N> failures (<verdicts>)
Iterations: <N>/<max>
Surface: <surface-file-path>
Plan: .agent/{branch}/test-auto-fix-plan.md
Escalation reason: <…> # if not green
Self-Improvement
/test-auto-fix gets better across runs through a two-tier lessons loop (fast
episodic tier + gated promotion), like autonomous-workflow and fix-bug. It
reads test-auto-fix-lessons at Phase 2 (biasing the verdict and the Phase 3
fix sub-class) and writes at Phase 6 (regression / same-failure signal) and
Phase 7 (outcome retrospective), keyed by stack : failure-pattern : verdict-sub-class. This complements the surface file (which is config — how
to run tests here — not learned judgment) rather than duplicating it. Lessons are
advisory — they never lower the confidence gate, never turn a
prod-bug/unsure escalation into a silent test edit, and never override a
refusal in rules/anti-patterns.md.
Most value is within a project — the repo::{owner}/{repo} scope (catching a
recurring verdict misclassification for a failure shape); cross-project leverage
(the global scope) is weaker because the feedback is binary and local. A
recurring lesson (seen_count >= 3) is promotion-eligible via
/create-skill diagnose test-auto-fix. Lessons run through LoreKit's memory.*
tools (the lorekit-memory skill); if LoreKit is not connected the loop is a
silent no-op. Full contract:
rules/self-improvement-loop.md.
Definition of done
The run is done when ANY of the following is true:
- All surface tests are green AND the structured exit summary has been printed.
- All remaining failures are
prod-bug or unsure (escalated; user owns next step).
- The confidence gate scored < 80 and the fix was not written.
- A regression was detected, reverted, and the user owns the next step.
--max-iterations (default 5) was reached.
