| name | test-red-team |
| description | Launch an adversarial reviewer (Opus) to find bugs IN THE TEST SUITE — tautological assertions, stub lies, jsdom-shim divergence, vacuous system checks, coverage gaps. Use when the user says "red team the tests", "audit the tests", "are our tests any good", or after adding a large batch of tests. |
Test-suite red-team review
Launches an Opus general-purpose agent to hunt bugs in the tests
themselves, not the production code. goldseam's whole pitch is trust —
a heal pipeline whose own tests lie is worse than no tests. Complementary
to red-team-review (prod code) and
mutation-red-team (dynamic check); run
all three at major change points.
When to invoke
- User says: "red team the tests", "audit the tests", "find weak tests",
"are our tests rigorous"
- After adding a large batch of tests
- After a red-team-review finds a bug the suite missed — ask "why didn't
our tests find this?"
- Before publishing any release
How to invoke
Use the Agent tool with:
subagent_type: "general-purpose"model: "opus"description: 3–5 word description (e.g. "Red-team test suite")prompt: follow the template below
Prompt template
Fill the bracketed sections with current project state. Do NOT send as-is.
You are an adversarial code reviewer performing a red-team bug hunt on
the TEST SUITE of goldseam (Cypress self-healing plugin + CLI). Your
target is the correctness and rigor of the tests — not the production
code. Rank findings CRITICAL / HIGH / MEDIUM / LOW.
## Setup
Project: the goldseam repo root (`git rev-parse --show-toplevel`).
Read AGENTS.md §"Tests + build" for the suites:
- packages/goldseam/test/*.test.ts (vitest; jsdom for browser-side)
- scripts/system-test.mjs, scripts/hardening-test.mjs,
scripts/heal-e2e-test.mjs, scripts/prompt-e2e-test.mjs (Module-API
system suites; check() helpers)
You may run them (npm run build:packages first; suites start their own
demo server on 4173).
## What's worth hunting — the "how tests lie" checklist
1. Tautological assertions: for each test, ask "if the production code
were replaced with a no-op / return [] / always-pass, would this test
fail?" If no, it is vacuous.
2. Stub-Cypress lies: support-invariants.test.ts reimplements
Cypress.on/listeners/currentRetry and mocha hooks. Where does the
stub's semantics diverge from real Cypress 15 (listener ordering,
throw propagation between listeners, afterEach `this` binding,
retries())? A test passing against the stub proves nothing where the
stub diverges.
3. jsdom lies: the pseudo-element getComputedStyle shim, cloneNode
fidelity, TreeWalker semantics — do redaction tests pass for
jsdom-specific reasons?
4. Vacuous system checks: in the three .mjs suites, do any check()
calls assert something that cannot fail (wrong variable, always-true
condition, checking existence of a file the suite itself created)?
5. Stub-model circularity: heal-e2e proves the engine end-to-end, but
the stub emits the known-correct edit. Which failure modes of a real
model does the E2E therefore never exercise? Is anything asserting
the rerun rungs can actually REJECT a bad edit?
6. Coverage gaps: production branches no test exercises (engine error
paths, CLI arg handling, runner failures, artifact-write failures,
maxDomBytes truncation, ariaSnapshot: false).
7. Shared-state leakage: module-level `installed` guard + listener
arrays in unit tests; port 4173 collisions between suites; .goldseam
dir cleanup between scenarios.
8. Assertion granularity: length-only or existence-only assertions where
content matters (artifact fields, ladder verdicts, redaction output).
## Output
300–600 words. Severity-grouped. For each finding:
- file:line reference
- one-line description of why the test is weak
- concrete trigger: "an implementation that does X would still pass"
No fixes — diagnose only. "No CRITICAL issues found" is valuable signal;
say so explicitly.
Scripts and assets
No scripts today. Coverage helpers, if ever needed, live in this folder.
Cleanup discipline
Same rules as red-team-review: the agent
is read-only; no repo files; response inline; final git status check.
Confirmed gaps become tests (preferred) or entries in the deferred-findings
ledger (invariants-and-tradeoffs.md)
when they are accepted tradeoffs — never silent knowledge.
