test-quality-guard

name

test-quality-guard

description

Kills fake, low-value, or bug-locking tests — tests that assert nothing meaningful, tests that mock the very unit under test, and tests that assert CURRENT (possibly buggy) behavior instead of correct behavior. Best used reactively after tests are written, generated, or changed. Use when the user says "add tests", "write unit tests", "increase coverage", "is this tested?", or after a coding agent produced or modified test code. DO NOT USE for production (non-test) code review (use clean-diff-guard) or for verifying a task is done (use evidence-before-done); this guard reviews TEST code quality specifically.

test-quality-guard

Passing tests are not proof of a good test. A test that asserts nothing, mocks the code it's supposed to check, or asserts today's buggy output will pass — and pass forever, even after the bug is "fixed" or the logic is deleted. Roughly half of AI-generated unit tests assert nothing meaningful, and Meta's TestGen-LLM study found only ~25% of generated tests actually increased real coverage. This guard exists to catch the other 50-75%.

When to use — three modes

• Guard-pass (default). Right after tests are written, generated, or edited — by you or by another agent turn — before presenting them as done. Review the diff of test files before moving on.
• Live. While writing tests interactively: as each test is drafted, ask "what observable behavior does this assertion actually pin down?" before moving to the next one. Don't batch a pile of untested tests for a final review.
• Review. When the user asks "is this tested?" / "are these good tests?" / "why did this bug ship despite 95% coverage?": audit existing test files as a critic. Report which tests are real and which are decorative. Do not silently rewrite tests unless asked to fix them.

What this guard blocks

1. No meaningful assertion. A test that only calls the function under test, or that asserts toBeDefined() / toBeTruthy() / not.toBeNull() on a mock's return value, proves nothing about behavior — FAIL. A real test asserts a specific expected value, error, or state change tied to the input.
2. Mocking the unit under test. Only mock true external boundaries — network calls, the system clock, the filesystem, a third-party API, a database client. If the test mocks the very function, class, or module it claims to test, it is testing the mock, not the code — FAIL. Mock one layer below the unit under test, never the unit itself.
3. Asserting current (buggy) behavior instead of correct behavior. If the task was to FIX a bug, the test must assert what the CORRECT output should be, not what the current code happens to produce. Never write expect(divide(10, 0)).toBe(0) to make a buggy implementation pass — that locks the bug in as "expected behavior" forever.
4. Missing fail-before/pass-after for bug fixes. A bug-fix test is only evidence of the fix if it FAILS against the pre-fix code and PASSES against the post-fix code. If you can't show (or reason through) that the test would have caught the original bug, say so explicitly — don't just assert the new test "looks right."
5. Duplicate near-identical test bodies. Multiple tests that differ only in cosmetic input values but exercise the exact same code path and assertion shape pad the test count without adding coverage of a new behavior, edge case, or branch. Each test should earn its place by covering something the others don't.
6. Implementation-detail assertions over behavior/contract assertions. Prefer asserting on observable outputs, return values, thrown errors, or state changes a caller would see. Asserting that a private helper was called N times, or a mock's internal call args, couples the test to the implementation and breaks on harmless refactors while missing actual behavior regressions.
7. Coverage % is not evidence of quality. A file can hit 100% line coverage while every test is a no-op (see rule 1) or a mocked-out unit (see rule 2). Do not accept a coverage number as proof that the tests are good — flag "coverage theater" and check what the coverage is actually backed by.

Procedure

1. Identify the unit under test. Name the exact function, method, or module the test file claims to cover, and its real dependencies (network, DB, clock, fs, other modules).
2. Check what's mocked. For each mock/stub/spy in the test file, confirm it targets a true external boundary, not the unit under test itself or a sibling function that's part of the same logical unit.
3. Check each test has a real assertion. For every test/it block, find the assertion(s) and confirm each one is tied to an observable, specific value or behavior — not a truthy/defined/no-throw check used as a stand-in for a real expectation.
4. For bug fixes, verify fail-before/pass-after. Read (or mentally trace) the pre-fix code against the new test. If the test would have passed on the buggy code too, it is not a regression test — it's decoration. State explicitly whether the fail-before/pass-after property holds.
5. Scan for duplicates and implementation-detail coupling. Flag near- identical test bodies and any assertion on internal call mechanics rather than outward behavior.
6. Decide and report. If any of the 7 items above are found, list each with the offending test name and the concrete fix (what assertion to add or change). If the tests are genuinely sound, say so plainly — do not invent findings to look thorough.
7. Emit the guard footer (see Output) as the last line of the response.

Output

Emit this verbatim as the last line of any response where test code was written, generated, modified, or reviewed:

proofguard:test-quality-guard — <PASS | FIX-REQUIRED | WAIVED(reason)> · Triggered: <what fired it> · Fixed: <n> · Verified: <what was checked — mocks, assertions, fail-before/pass-after> · Remaining gap: <what's still unchecked or accepted as-is>

• PASS — every test has a real assertion, mocks stay at the true boundary, and any bug-fix test provably fails before / passes after.
• FIX-REQUIRED — one or more of the 7 blocked patterns was found; the specific test(s) and fix are listed above the footer.
• WAIVED(reason) — a flagged pattern is being knowingly accepted (e.g. a smoke test with a deliberately loose assertion), and that tradeoff has been stated to the user, not glossed over.

References

• references/test-smells.md — catalog of the six test smells (no-assert, mock-the-unit, assert-current-behavior, duplicate bodies, implementation-detail assertions, coverage theater) with bad/good examples in JS and Python.
• references/behavior-first-assertions.md — how to write behavior-first assertions, and the fail-before/pass-after rule for bug-fix tests in detail.

What this guard does NOT do

It does not run the test suite or judge whether the overall task is done — that's evidence-before-done (which also verifies that a bug-fix check fails-before/passes-after at the task level). It does not review production (non-test) code for scope, style, or correctness — that's clean-diff-guard. This guard's only job is judging whether the TEST CODE ITSELF is real evidence of correct behavior, or decoration that happens to pass.