| name | test-audit |
| description | Sample ottochain test suites and prove they actually test — mutate each suite's subject, confirm the test goes red, then revert. Files issues for decorative/tautological/self-regenerating tests. Use to check test integrity, not just coverage. |
test-audit
Coverage % says a line ran; it does not say a test would catch a bug. This skill proves it the only way
that counts: break the code the test covers and confirm the test fails. Applies the
ai-smells-test-integrity persona checklist.
Sequence
bin/worksheet test-audit-<date> — open a worksheet for findings.- Pick a sample of suites — prioritize the guards named in
docs/signing-canonical-and-validation.md
and any suite touched by recent diffs. Good starting targets on main: CommittedViewSuite,
signing-canonical suites, FailureReasonCanonicalSuite, genesis contract suites.
- Confirm each suite is GREEN as-is:
bin/test <module> "*<SuiteName>*".
- The mutation check (the core of this skill): for each suite, make a small, targeted mutation to the
PRODUCTION code it covers — flip a comparator (
>→>=), negate a boolean, return a constant, drop a
sortBy, skip a dedup. Re-run the suite:
- Suite goes RED → the test is real. Good. Revert the mutation (
git checkout -- <file>).
- Suite stays GREEN → the test is decorative for that behavior. Record it as a finding, revert.
ALWAYS revert every mutation before moving on — leave the tree clean.
- Apply the rest of the persona checklist by reading:
- self-regenerating golden fixtures (fixture written and asserted in the same run),
- tautological assertions (asserting a mock returns what it was told),
- vacuous property generators, error-message-string drift, weaver
name shadowing.
- File a
gh issue per weak suite (title lowercase ≤72, label test-integrity), citing the mutation
that survived. Link findings in the worksheet. Append a docs/agents/feedback.md entry if a whole
class of behavior is untested.
Rules
- Never leave a mutation in the tree. If interrupted,
git status + revert before stopping.
- Do not "fix" the weak test inline during the audit — file it; the fix is a separate reviewed PR.
- A suite that goes red under every plausible mutation is a PASS — record that too, it's signal.
Future backbone
This is manual mutation testing. The durable version is stryker4s (mutation testing for Scala) run in
CI — a mutation score is a deterministic, non-flaky "do the tests catch bugs" gate, unlike the advisory
70% coverage number. A stryker4s spike is proposed in docs/agents/process-observations.md (#4); until it
lands, this skill is how we get the signal.
