mutation-red-team

name

mutation-red-team

description

Inline mutation testing for the apps/ monorepo — inject targeted regressions into @iep/domain (machine guards, rights derivation), @iep/data (Drizzle queries, schema), @iep/services (mutation contracts, composition), or apps/web (server actions), then run `pnpm -w typecheck` + `pnpm -w test` and report which mutations SURVIVED. Surviving mutations are direct evidence of validation gaps. Runs inline in the current chat (no subagent fork). Use when the user says "trickster", "mutation test", "break the code", "can the tests catch regressions", "grade the validation", or after adding code whose coverage is uncertain.

Mutation red-team (the trickster, inline)

Run inline in the current chat using the session's own Read, Edit, and Bash tools. No Agent fork, no isolated worktree — the main session edits production code, runs validation, then reverts with git checkout -- <file>. The working tree must be clean before starting and clean after each mutation.

Surviving mutations are the finding. A SURVIVED mutation means the validation surface could not distinguish broken code from working code — a concrete coverage gap pointing at a specific invariant nothing actually enforces.

Complementary to:

• test-red-team — reads the validation surface statically.
• red-team-review — reads production code statically.
• citation-audit — audits user-facing legal citations against the OBL register.

This is the dynamic, empirical check: does the validation surface actually catch bad code.

For context on why mutation scores beat coverage metrics, see Test Double's "Keep your coding agent on task with mutation testing".

When to invoke

• User says: "trickster", "mutation test", "break the code", "grade the tests", "can my tests catch regressions"
• After red-team-review finds a prod bug — run a mutation against that invariant to confirm any new regression test actually catches future recurrences
• After a large lift (new state machine, new query module, new server action surface) — spot-check coverage on the load-bearing invariants
• Proactively on the load-bearing invariants below
• Do NOT run alongside another test invocation in the same shell — the workspace test runs in parallel and mutation reverts must be serialized

Pre-flight (mandatory)

1. git status — must be clean. If dirty, ask the user to commit/stash first.
2. docker ps --format '{{.Names}}' | grep -q iep-app-db — integration tests need the DB. If missing, pnpm db:up first.
3. Record the file you're about to touch: git log -1 --format='%h %s' -- <file> so the revert target is unambiguous.

Curated mutations (this project)

Do NOT pick random lines — signal-to-noise is terrible for random mutants. Hand-pick from load-bearing invariants. Starter set, organized by package:

@iep/domain/machines/ard

1. quorumGate excusal off — in machine.ts, in the quorum gate's loop over requiredRoles, change if (excusedWithInput) return true; to return false; (or delete the excused branch). A required role that's excused-with-input now blocks convening. Tests under machine.test.ts and participant-candidates.test.ts must catch the "convene fails when excusal should cover" case.

2. recordingGate special-factors off — drop the REQUIRED_SPECIAL_FACTORS.every(...) check from recordingGate. Recording transition now fires even when no special-factors data was considered. finalization-gate.test.ts and the recording branch of machine.test.ts must catch it.

3. recessAllowed carve-out invert — flip !(context.activePhysicalDanger || context.expellableOffense) to (context.activePhysicalDanger || context.expellableOffense). Recess is now allowed only when the carve-out should block it. Scenarios recess-blocked-campus-danger and recess-blocked-expellable-offense must catch it on replay.

4. amendmentInScope red-zone leak — in amendment-scope.ts, change isGreenZone(field.kind) to true. Every amendment is now in scope, including placement / eligibility changes that require a full ARD. amendment-scope-rejected-placement scenario plus amendment-scope.ts tests must catch it.

5. Goal validator field drop — in goal-validator.ts, remove baseline from GOAL_FIELDS. The OBL-011 finalization gate now passes goals missing a baseline. goal-validator.test.ts must catch it.

@iep/domain/rights

6. Role-grant window off — in rights/derive.ts (or wherever loadActiveEdges filters), remove the startAt <= now predicate so future-dated grants are active immediately. derive.test.ts must catch it via a "future grant is not yet active" assertion.

7. canPerform admin escape hatch — in can-perform.ts, add an unconditional if (persona.kind === "administrator") return true; at the top. Admins now bypass every action-level check, including the ones meant to enforce a separation of duties. can-perform.test.ts should have at least one "admin cannot perform parent-only action" test that fails.

@iep/data/queries

8. inArray empty-list bug — pick any query that uses inArray(col, ids) (e.g. iepQueries.listIepsForStudents) and remove the early-return when ids.length === 0. Drizzle's inArray([]) returns no rows — calling with an empty selection now returns nothing instead of nothing-by-design. Integration test must catch the "empty input → empty output" case.

9. Time-window flip — in any query that filters where(...) by (startAt <= now AND (endAt IS NULL OR endAt > now)), flip endAt > now to endAt >= now. Edge case at the exact tick. Hard to catch from validation; flagging SURVIVED here surfaces the absence of a "boundary-tick" integration test.

apps/web

10. Server-action persona check drop — pick any actions.ts that calls requirePersonaKind(persona, [...]) and remove it. An unauthorized persona can now invoke the action. Unit tests rarely cover this; the gap is the lack of action-level auth assertions.

11. revalidatePath drop — remove the revalidatePath(...) line from any server action. The mutation succeeds but the page doesn't update. Likely SURVIVED today; flagging it surfaces the absence of e2e cache-invalidation tests.

12. Audit emission drop — find an action that calls services.audit.emit(...) and delete the call. The state mutation succeeds; the audit row never lands. Tests under audit.integration.test.ts should catch it for the actions covered there. SURVIVED on an uncovered action is a coverage gap.

Each targets a named invariant or prior bug class. A SURVIVED result on any of them is a specific, actionable coverage gap.

Procedure (one mutation per cycle)

For each mutation:

1. State the mutation. Print the file, the exact before → after change, and one sentence on which invariant it probes.
2. Apply. Use Edit to make exactly the specified change. Do not touch test files, fixtures, schema, or seed data — only the named production file.
3. Run validation. From the repo root:

   pnpm -w typecheck && pnpm -w test
   

   Capture stdout+stderr. Note which step failed (if any) and which test name fired.
4. Conditional probes.
   • If the mutation is in @iep/data, ensure the Docker postgres is up — integration tests silently skip if DATABASE_URL is missing.
   • If the mutation is in apps/web, also run pnpm --filter web build — Next.js's compile-time type checking sometimes catches contract drift the workspace tsc misses.
5. Revert. git checkout -- <file>. Re-run git status; it must be clean.
6. Verdict.
   • CAUGHT if any validation step failed after the mutation was applied. Name the failing step + specific test. One to three sentences — is the failure specific to the invariant, or incidental?
   • SURVIVED if all steps passed. State what the mutation means the code now does incorrectly, and speculate on what kind of test would have caught it (a domain unit test? a Drizzle integration test? a server-action test? a guard-branch fixture?). No fix — diagnosis only.

Run one mutation per cycle. If covering multiple, run them sequentially with a clean tree between each.

Output

Inline per mutation, ~150–300 words. Lead with the one-word verdict. Then the failing-step name (if CAUGHT) or the coverage-gap description (if SURVIVED).

After all mutations:

• Mutation score = CAUGHT / total.
• Surviving mutations by invariant — every SURVIVED maps to one regression test or runtime check that should be added.
• Equivalent mutants — call these out separately; they don't count as failures of the validation surface.

Cleanup

Stricter than the static red-team skills because you write code.

• After every mutation: git checkout -- <file> and verify git status is clean before starting the next mutation.
• No scratch files. No prompt templates left behind.
• Mutation output belongs inline in the conversation, not in a .md file in the repo. SURVIVED findings may be long — condense before reporting, don't dump.
• Before returning control: git status must be clean; no stray edits, no committed mutations.