test-red-team

name	test-red-team
description	Launch a skeptical reviewer (Opus) to find bugs IN THE TEST SUITE — tautological assertions, mock/stub divergence from reality, blindly-regenerated golden snapshots, vacuous invariance checks, coverage gaps. Use when the user says "red team the tests", "audit the tests", "are our tests any good", or after adding a batch of tests.

Test-suite bug hunt

A skeptical review that hunts bugs in the tests themselves, not the production code. A suite whose own tests lie is worse than no tests — it manufactures false confidence. The invoking agent supplies this project's specifics from AGENTS.md. Complementary to red-team-review (prod code + pipeline) and mutation-red-team (dynamic check); run all three at major change points.

When to invoke

• User says: "red team the tests", "audit the tests", "find weak tests", "are our tests rigorous"
• After adding a batch of tests
• After a red-team-review finds a bug the suite missed — ask "why didn't our tests find this?"
• Before publishing any release

How to invoke

Use the Agent tool with:

• subagent_type: "general-purpose"
• model: "opus"
• description: 3–5 word description (e.g. "Red-team test suite")
• prompt: fill the bracketed sections below — do NOT send as-is.

Prompt template

You are a skeptical code reviewer performing a break-it bug hunt on the TEST
SUITE. Your target is the correctness and rigor of the tests — not the
production code. Rank findings CRITICAL / HIGH / MEDIUM / LOW.

## Setup

The repo root (`git rev-parse --show-toplevel`). Read AGENTS.md §Tests+build
for the policy (static green floor: types, lint, unit tests). The web app
uses `npm run check` (svelte-check + tsc) in web/; unit tests live wherever
this increment pinned them. Run what you can.

[Name the suites and where they live; the invariants they defend (from
AGENTS.md — e.g. the GPS publish gate, slug identity, manifest/data.ts
schema agreement, projection/clustering math, deterministic rebuild); and
what is deliberately NOT unit-tested (anything touching a live browser
canvas, exiftool/sips/ffmpeg on real media, or network) so you judge the
tests around those seams, not their absence.]

## What's worth hunting — the "how tests lie" checklist

1. Tautological assertions: for each test, ask "if the production code were
   replaced with a no-op / return [] / always-pass, would this test fail?"
   If no, it is vacuous. (E.g. a GPS-gate test that never feeds it a file
   with surviving location metadata.)
2. Mock/stub divergence: tests that stub an external dependency (exiftool,
   sips, ffmpeg, fetch, the DOM/canvas). Where does the stub's semantics
   diverge from the real thing? A pass against a wrong stub proves nothing.
3. Blindly-regenerated goldens/snapshots: golden-file or snapshot tests
   regenerated from CURRENT behavior without a human reading the diff — they
   pin whatever was captured, bug included. Which assert an actual invariant
   vs merely echo output?
4. Vacuous invariance checks: a property test (determinism, idempotency,
   round-trip rebuild) that exercises the property only ONCE, or against
   shared cached/seeded state, so it cannot catch the dependency it claims
   to rule out.
5. Untrusted captures: a test that trusts an artifact (screenshot, rendered
   derivative, manifest) without a validity gate — asserting on a broken or
   half-formed output as if it were good.
6. Coverage gaps: production branches no test exercises — error paths, the
   GPS-survived exit path, slug-collision assert, boundary/limit
   enforcement, recovery/cleanup paths.
7. Shared-state leakage: temp dirs, fixtures, or global state not isolated
   or cleaned between tests, so order or a stale artifact changes the result.
8. Assertion granularity: length-only or existence-only assertions where
   content matters (e.g. asserting a manifest has N entries but not that
   coordinates/slugs are correct).

## Output

300–600 words. Severity-grouped. For each finding:
- file:line reference
- one-line description of why the test is weak
- concrete trigger: "an implementation that does X would still pass"

No fixes — diagnose only. "No CRITICAL issues found" is valuable signal; say
so explicitly.

Scripts and assets

No scripts. Coverage helpers, if ever needed, live in this folder.

Cleanup discipline

Same rules as red-team-review: the agent is read-only; no repo files; response inline; final git status check. Confirmed gaps become tests (preferred) or a recorded accepted-tradeoff where the knowledge lives — never silent knowledge.