test-red-team

name

test-red-team

description

Adversarial review of the TEST AND VALIDATION SURFACE of the apps/ monorepo — vitest unit suites in @iep/domain + @iep/data + @iep/services + @iep/ui + apps/web, the Drizzle integration tests under @iep/data/queries, the boundary.test.ts pattern enforcers in @iep/ui and @iep/data, and the OBL enforcement-map. Hunts tautological assertions, integration tests that don't reach the DB, missing regression coverage for known-deferred obligations, and surfaces that document an invariant without enforcing it. Runs inline in the current chat (no subagent fork). Use when the user says "red team the tests", "audit the tests", "are our tests any good", or after adding tests/validation that needs scrutiny.

Test-suite red-team review (inline)

Hunt weakness in the validation surface, not the production code. Run inline using the session's own Read, Grep, and Bash tools. Do NOT spawn an Agent.

The reality check is whatever pnpm test actually exercises today — re-discover at invocation time, don't assume a fixed shape. Run the Setup block first so your reasoning reflects current state.

Complementary to red-team-review (production-code bugs), mutation-red-team (empirical coverage), and citation-audit (legal-grounding audit of user-facing cites).

When to invoke

• User says: "red team the tests", "audit the tests", "find weak tests", "are our tests rigorous", "test coverage gaps"
• After adding any new test file
• After red-team-review finds a bug nothing caught — ask "why didn't anything find this?"
• Before relying on the test suite to gate a deploy

Setup — discover the current validation surface

# What test files exist?
find packages apps -name "*.test.ts" -o -name "*.test.tsx" -o -name "*.integration.test.ts" \
  | grep -v node_modules | sort

# Are there boundary-style pattern enforcers?
ls packages/*/src/boundary.test.ts 2>/dev/null

# What does `pnpm test` invoke per package?
grep -E '"test"' packages/*/package.json apps/*/package.json

# Are the integration tests actually wired to a DB?
grep -l "createPostgresClient\|drizzle\|pg-client" packages/data/src/queries/*.integration.test.ts | head

# Is the DB up? Integration tests silently no-op if it isn't.
pnpm db:up >/dev/null 2>&1 && docker ps --format '{{.Names}}' | grep -q iep-app-db && echo "db: up" || echo "db: DOWN — integration tests cannot run"

The output tells you what's mechanized vs. documented but unenforced.

What to hunt — the "how validation lies" checklist

Rank findings CRITICAL / HIGH / MEDIUM / LOW.

1. Tautological assertions. Pick each assertion and ask: "if the production code returned an empty array / undefined / a constant, would this still pass?" If yes, the check is vacuous. Common shapes:

   • expect(rows).toBeDefined() after a query that always returns an array.
   • expect(rows.length).toBeGreaterThanOrEqual(0) — every array satisfies this.
   • expect(result).not.toThrow() without inspecting the result.

2. Integration tests that don't actually integrate. packages/data/src/queries/*.integration.test.ts need a live Postgres. If the test file describe.skips on a missing DATABASE_URL, or silently passes when the DB is down, the suite green-lights queries that have never run. Check the test setup for if (!process.env.DATABASE_URL) return; early returns.

3. Boundary test bypass. packages/ui/src/boundary.test.ts and packages/data/src/boundary.test.ts are grep-based pattern enforcers. Look for .filter((line) => !line.startsWith(...)) allowances — every exception is a permission slip. Is the exception still load-bearing or stale? An allowlist for a deleted file masks future drift in a file that should have been re-added.

4. Pure unit tests where an integration would catch more. @iep/domain/machines/ard/machine.test.ts exercises the machine in isolation. The replay path that actually runs in the app calls replayARDLifecycle over audit_logs joined to meetings — that path is only covered indirectly. A bug between the query's row shape and AuditReplayEvent would slip past the domain test.

5. Schema tests assert shape, not constraints. packages/data/src/schema/schema.test.ts enumerates column names and types. Does it assert NOT NULL where it matters? Does it assert FK cascade rules (onDelete: "cascade" vs. "set null")? Does it assert default values for participants: '[]'::jsonb so a hand-edit doesn't silently relax the contract?

6. Time-window logic untested. @iep/domain/rights filters role assignments by startAt/endAt. Is there a test that an expired role-grant disappears from ? A test that a grant doesn't activate yet? Missing both = the time window is unenforced.

Output

Inline, 300–600 words. Severity-grouped. For each finding:

• file:line reference (markdown link form [name](path#Lline)) or MISSING: <test description> for gaps.
• one-line description of why the validation is weak / absent.
• concrete trigger: "an implementation that does X would still pass / go undetected".

No fixes — diagnose only. "No CRITICAL issues found" is valuable signal when warranted; say so explicitly.

Cleanup

You are the main session — output findings inline, no scratch files. End with a git status check.