| name | test-red-team |
| description | Launch a skeptical reviewer (Opus) to find bugs IN THE TEST SUITE — tautological assertions, mock/stub divergence from reality, blindly-regenerated golden snapshots, vacuous invariance checks, coverage gaps. Use when the user says "red team the tests", "audit the tests", "are our tests any good", or after adding a batch of tests. |
Test-suite bug hunt
A skeptical review that hunts bugs in the tests themselves, not the
production code. A suite whose own tests lie is worse than no tests — it
manufactures false confidence. The invoking agent supplies this project's
specifics from AGENTS.md. Complementary to
red-team-review (prod code + pipeline) and
mutation-red-team (dynamic check); run all
three at major change points.
When to invoke
- User says: "red team the tests", "audit the tests", "find weak tests",
"are our tests rigorous"
- After adding a batch of tests
- After a red-team-review finds a bug the suite missed — ask "why didn't
our tests find this?"
- Before publishing any release
How to invoke
Use the Agent tool with:
subagent_type: "general-purpose"model: "opus"description: 3–5 word description (e.g. "Red-team test suite")prompt: fill the bracketed sections below — do NOT send as-is.
Prompt template
You are a skeptical code reviewer performing a break-it bug hunt on the TEST
SUITE. Your target is the correctness and rigor of the tests — not the
production code. Rank findings CRITICAL / HIGH / MEDIUM / LOW.
## Setup
The repo root (`git rev-parse --show-toplevel`). Read AGENTS.md §Tests+build
for the policy (static green floor: types, lint, unit tests). The web app
uses `npm run check` (svelte-check + tsc) in web/; unit tests live wherever
this increment pinned them. Run what you can.
[Name the suites and where they live; the invariants they defend (from
AGENTS.md — e.g. the GPS publish gate, slug identity, manifest/data.ts
schema agreement, projection/clustering math, deterministic rebuild); and
what is deliberately NOT unit-tested (anything touching a live browser
canvas, exiftool/sips/ffmpeg on real media, or network) so you judge the
tests around those seams, not their absence.]
## What's worth hunting — the "how tests lie" checklist
1. Tautological assertions: for each test, ask "if the production code were
replaced with a no-op / return [] / always-pass, would this test fail?"
If no, it is vacuous. (E.g. a GPS-gate test that never feeds it a file
with surviving location metadata.)
2. Mock/stub divergence: tests that stub an external dependency (exiftool,
sips, ffmpeg, fetch, the DOM/canvas). Where does the stub's semantics
diverge from the real thing? A pass against a wrong stub proves nothing.
3. Blindly-regenerated goldens/snapshots: golden-file or snapshot tests
regenerated from CURRENT behavior without a human reading the diff — they
pin whatever was captured, bug included. Which assert an actual invariant
vs merely echo output?
4. Vacuous invariance checks: a property test (determinism, idempotency,
round-trip rebuild) that exercises the property only ONCE, or against
shared cached/seeded state, so it cannot catch the dependency it claims
to rule out.
5. Untrusted captures: a test that trusts an artifact (screenshot, rendered
derivative, manifest) without a validity gate — asserting on a broken or
half-formed output as if it were good.
6. Coverage gaps: production branches no test exercises — error paths, the
GPS-survived exit path, slug-collision assert, boundary/limit
enforcement, recovery/cleanup paths.
7. Shared-state leakage: temp dirs, fixtures, or global state not isolated
or cleaned between tests, so order or a stale artifact changes the result.
8. Assertion granularity: length-only or existence-only assertions where
content matters (e.g. asserting a manifest has N entries but not that
coordinates/slugs are correct).
## Output
300–600 words. Severity-grouped. For each finding:
- file:line reference
- one-line description of why the test is weak
- concrete trigger: "an implementation that does X would still pass"
No fixes — diagnose only. "No CRITICAL issues found" is valuable signal; say
so explicitly.
Scripts and assets
No scripts. Coverage helpers, if ever needed, live in this folder.
Cleanup discipline
Same rules as red-team-review: the agent is
read-only; no repo files; response inline; final git status check.
Confirmed gaps become tests (preferred) or a recorded accepted-tradeoff
where the knowledge lives — never silent knowledge.
