test-red-team

name

test-red-team

description

Adversarial review of the TEST AND VALIDATION SURFACE of the apps/ monorepo — vitest unit suites in @iep/domain + @iep/data + @iep/services + @iep/ui + apps/web, the Drizzle integration tests under @iep/data/queries, the boundary.test.ts pattern enforcers in @iep/ui and @iep/data, and the OBL enforcement-map. Hunts tautological assertions, integration tests that don't reach the DB, missing regression coverage for known-deferred obligations, and surfaces that document an invariant without enforcing it. Runs inline in the current chat (no subagent fork). Use when the user says "red team the tests", "audit the tests", "are our tests any good", or after adding tests/validation that needs scrutiny.

Test-suite red-team review (inline)

Hunt weakness in the validation surface, not the production code. Run inline using the session's own Read, Grep, and Bash tools. Do NOT spawn an Agent.

The reality check is whatever pnpm test actually exercises today — re-discover at invocation time, don't assume a fixed shape. Run the Setup block first so your reasoning reflects current state.

Complementary to red-team-review (production-code bugs), mutation-red-team (empirical coverage), and citation-audit (legal-grounding audit of user-facing cites).

When to invoke

• User says: "red team the tests", "audit the tests", "find weak tests", "are our tests rigorous", "test coverage gaps"
• After adding any new test file
• After red-team-review finds a bug nothing caught — ask "why didn't anything find this?"
• Before relying on the test suite to gate a deploy

Setup — discover the current validation surface

# What test files exist?
find packages apps -name "*.test.ts" -o -name "*.test.tsx" -o -name "*.integration.test.ts" \
  | grep -v node_modules | sort

# Are there boundary-style pattern enforcers?
ls packages/*/src/boundary.test.ts 2>/dev/null

# What does `pnpm test` invoke per package?
grep -E '"test"' packages/*/package.json apps/*/package.json

# Are the integration tests actually wired to a DB?
grep -l "createPostgresClient\|drizzle\|pg-client" packages/data/src/queries/*.integration.test.ts | head

# Is the DB up? Integration tests silently no-op if it isn't.
pnpm db:up >/dev/null 2>&1 && docker ps --format '{{.Names}}' | grep -q iep-app-db && echo "db: up" || echo "db: DOWN — integration tests cannot run"

The output tells you what's mechanized vs. documented but unenforced.

What to hunt — the "how validation lies" checklist

Rank findings CRITICAL / HIGH / MEDIUM / LOW.

1. Tautological assertions. Pick each assertion and ask: "if the production code returned an empty array / undefined / a constant, would this still pass?" If yes, the check is vacuous. Common shapes:

   • expect(rows).toBeDefined() after a query that always returns an array.
   • expect(rows.length).toBeGreaterThanOrEqual(0) — every array satisfies this.
   • expect(result).not.toThrow() without inspecting the result.

2. Integration tests that don't actually integrate. packages/data/src/queries/*.integration.test.ts need a live Postgres. If the test file describe.skips on a missing DATABASE_URL, or silently passes when the DB is down, the suite green-lights queries that have never run. Check the test setup for if (!process.env.DATABASE_URL) return; early returns.

3. Boundary test bypass. packages/ui/src/boundary.test.ts and packages/data/src/boundary.test.ts are grep-based pattern enforcers. Look for .filter((line) => !line.startsWith(...)) allowances — every exception is a permission slip. Is the exception still load-bearing or stale? An allowlist for a deleted file masks future drift in a file that should have been re-added.

4. Pure unit tests where an integration would catch more. @iep/domain/machines/ard/machine.test.ts exercises the machine in isolation. The replay path that actually runs in the app calls replayARDLifecycle over audit_logs joined to meetings — that path is only covered indirectly. A bug between the query's row shape and AuditReplayEvent would slip past the domain test.

5. Schema tests assert shape, not constraints. packages/data/src/schema/schema.test.ts enumerates column names and types. Does it assert NOT NULL where it matters? Does it assert FK cascade rules (onDelete: "cascade" vs. "set null")? Does it assert default values for participants: '[]'::jsonb so a hand-edit doesn't silently relax the contract?

6. Time-window logic untested. @iep/domain/rights filters role assignments by startAt/endAt. Is there a test that an expired role-grant disappears from loadActiveEdges? A test that a future-dated grant doesn't activate yet? Missing both = the time window is unenforced.

7. State-machine guard coverage. quorumGate, recordingGate, recessAllowed, amendmentInScope — each guard's true and false branches deserve a test. Grep machine.test.ts for the guard name; if it appears only in the "happy path" describe block, the false branch is unverified.

8. OBL register vs. tests. docs/case-management-research/enforcement-map.md lists every IDEA / TEC / 19 TAC obligation and whether it's enforced. For each OBL marked ✅ or ⚠, is there a test that catches a regression of the enforcement? Most likely no — name the missing tests. For each ❌ row, the gap is known; that's not a test-suite failure, but flag any ❌ whose absence introduces a data-integrity risk vs. merely a UX gap.

9. Server-action tests are usually absent. apps/web/src/app/**/actions.ts rarely has direct unit tests. The compositional pattern means most of the logic moves to @iep/data and @iep/domain, but action-level concerns remain: requirePersonaKind, revalidatePath, audit emission, error envelope shape. Anything that asserts an action emits an audit row? Anything that asserts an unauthorized persona is rejected before mutation?

10. Composition tests are happy-path. packages/services/src/composition.test.ts (and the thin re-export under apps/web/src/lib/composition.ts) likely verifies buildServices(...) returns the expected service object. Does it verify that every method on the service object is actually wired (not undefined)? A typo'd composition key (compliace: ...) is the kind of bug a smoke test catches and a structural test misses.

11. Idempotent seed has no test. seed-personas.ts early-returns when an admin user already exists; seed-scenarios.ts uses onConflictDoNothing. Is there a test that running the seed twice produces the same row counts? That a partial prior run is detected and completed?

12. No regression test for known-deferred bugs. When red-team-review ships a "deferred" finding (e.g. TOCTOU in playNextStep), the commit message often notes the gap. Is there a corresponding test that would catch the recurrence if the bug were re-introduced after a refactor? Almost certainly no — name the missing tests.

Output

Inline, 300–600 words. Severity-grouped. For each finding:

• file:line reference (markdown link form [name](path#Lline)) or MISSING: <test description> for gaps.
• one-line description of why the validation is weak / absent.
• concrete trigger: "an implementation that does X would still pass / go undetected".

No fixes — diagnose only. "No CRITICAL issues found" is valuable signal when warranted; say so explicitly.

Cleanup

You are the main session — output findings inline, no scratch files. End with a git status check.