test-coverage

name	test-coverage
description	Perform a read-only audit of the codebase's test pyramid. Two halves of equal weight: (1) find critical flows lacking coverage; (2) audit *existing* tests for false confidence — shape-only assertions, tautological mocks, skipped/.only blocks, brittle selectors, tests that pass for the wrong reason. A green suite that doesn't exercise behavior is at least as dangerous as a missing one. Assumes Vitest for unit/integration and Playwright for e2e/smoke. Pragmatic by contract: does NOT chase 100% line coverage — focuses on logic, branching, money math, auth/tenant filters, and user-visible flows. Produces one ranked report with severity, finding kind (Gap vs Defect), evidence (file:line), and a concrete suggested fix. Use whenever the user asks for a "test audit", "test coverage review", "test pyramid health check", "audit our tests", "what should we test", or similar. Never modifies test code — read-only by contract.
allowed-tools	Bash(rg:*), Bash(grep:*), Bash(find:*), Bash(ls:*), Bash(wc:*), Bash(cat:*), Bash(sort:*), Bash(uniq:*), Bash(head:*), Bash(tail:*), Bash(awk:*), Bash(sed:*), Bash(jq:*), Bash(node:*), Bash(npm:*), Bash(pnpm:*), Bash(git:*), Read, Write

Test Coverage Audit

A read-only audit of the testing pyramid. The skill is methodology, not magic: walk the phases below, gather concrete file:line evidence of what is and is not tested, then emit one ranked report.

Operating contract

1. Read-only. Do not create or modify test files, fixtures, or config.
2. Evidence-first. Every finding cites path/to/file.ts:LINE for the production code at risk and (when relevant) the existing test that does or doesn't cover it. No vague claims like "tests should be expanded."
3. Audit existing tests, not just gaps. Half the skill's job is finding missing coverage; the other half is judging existing tests — shape- only assertions, over-mocking, snapshot rot, skipped/.only blocks, real network in unit tests, brittle e2e selectors, tests that pass for the wrong reason. A green suite that doesn't actually test behavior is worse than no suite, because it manufactures false confidence. Phase 5 is not optional — it's a peer of Phase 4.
4. Pragmatic, not exhaustive. The target is a healthy pyramid covering critical flows — NOT 100% line coverage. Skip trivial code (pure type helpers, single-line wrappers, shape-only DTOs). Focus on logic, branching, integrations, and user-visible flows.
5. Test logic, not shape. A "good test" exercises behavior under conditions that could plausibly fail. A test that asserts expect(obj). toHaveProperty('id') on a typed object is shape-only and should be flagged as low-value, not counted as coverage.

The pyramid

Use this mental model when judging health. Each layer answers a different question and has a different cost/speed/confidence tradeoff.

Layer	Question it answers	Typical tool	Cost
Unit	Does this function's logic do the right thing across its branches?	Vitest (node env)	Fast, cheap
Integration	Do these modules + the DB/queue/HTTP boundary cooperate correctly?	Vitest + real DB/queue or testcontainers	Medium
E2E	Can a real user complete a critical flow end-to-end in a browser?	Playwright	Slow, expensive
Smoke	After deploy, is the system minimally alive? (health, auth, one happy path)	Playwright (subset) or curl	Fast, run post-deploy

Healthy pyramid: many cheap unit tests at the base, fewer integration tests in the middle, a small set of e2e flows for happy + key error paths, a tiny set of smoke checks that run against production. Inverted pyramids (e2e-heavy) are slow and flaky; missing-middle pyramids (unit + e2e only) miss boundary bugs.

Severity rubric for findings

Findings come in two flavors: gaps (missing tests) and defects in existing tests (false confidence). Both use the same scale.

Severity	Gap example	Defect-in-existing example
Critical	A core revenue/data/security flow has no test of any kind.	An existing test for a critical flow asserts only shape / mocks every dependency / is permanently .skip'd — it appears green but verifies nothing.
High	Complex business logic has only happy-path coverage; error paths untested.	A test passes because it asserts against the same mock it set up (tautological); or covers only the happy path while claiming to cover the function.
Medium	An important flow is tested at the wrong layer (complex logic only via slow flaky e2e).	Stale snapshot covering rendered HTML that changes on every refactor; test uses Date.now() / real timers and is flaky in CI; test reaches live external services.
Low	Missing tests on non-critical helpers; duplication that increases maintenance cost.	Brittle CSS-selector-based e2e; over-broad mocking that hides regressions; redundant tests asserting the same property at multiple layers.
Info	Hygiene observation: configuration suggestions, no-runner-found.	A single stray .only left in; missing test naming convention; tests in the wrong directory.

A finding's severity reflects the cost of a regression slipping through, not the size of the fix. Bias toward the higher of likelihood and impact when they disagree. A passing test that doesn't actually exercise its target is at least as dangerous as a missing one — the missing test gets fixed when someone notices; the false-confidence test stays green forever.

Audit phases

Work these in order. Don't skip phases even if early ones look clean — different gaps surface in different phases.

Phase 1 — Recon

Build a map of what exists before judging what's missing.

# Repo shape & framework
find src -type d | head -60
cat package.json | jq '.scripts, .dependencies, .devDependencies'

# Test runners present?
cat package.json | jq '.scripts | to_entries | map(select(.value | test("vitest|playwright|jest|mocha"))) | .[]'

# Test config files
ls vitest.config* playwright.config* jest.config* 2>/dev/null

# All test files (broad net)
find . -type f \( -name '*.test.ts' -o -name '*.test.tsx' -o -name '*.spec.ts' -o -name '*.spec.tsx' -o -name '*.e2e.ts' \) -not -path '*/node_modules/*'

# Test directories
find . -type d \( -name '__tests__' -o -name 'tests' -o -name 'test' -o -name 'e2e' -o -name 'smoke' \) -not -path '*/node_modules/*'

# CI hooks for tests
cat .github/workflows/*.yml 2>/dev/null | rg -i 'test|vitest|playwright' -A2 -B1

Note: production framework, ORM, queue/worker, auth library, external integrations (LLM, email, payment, scraping). These are the boundaries where integration tests live.

Phase 2 — Map production surfaces to "what should be tested"

You can't judge missing tests without knowing what flows matter. Build a shortlist of critical flows by inspecting:

• Routes / endpoints. Each is a flow entry point.

  find src/routes src/app src/pages -type f 2>/dev/null
  rg -n 'createServerFn|defineHandler|export\s+(async\s+)?function\s+(GET|POST|PUT|DELETE|PATCH)' src/
  

• Jobs / workers. Each scheduled or queued job is a critical flow.

  find src/jobs src/worker -type f 2>/dev/null
  rg -n 'boss\.send|boss\.work|cron|schedule' src/
  

• Money/auth/data-isolation code paths. Pricing, billing, auth, tenant filters — high-cost-of-regression by definition.

  rg -n 'price|amount|cents|charge|invoice|stripe' src/
  rg -n 'auth|session|userId|orgId|tenantId' src/ | head -40
  

• External integrations. LLM/email/scraping adapters — boundary code that benefits most from integration tests.
• User-visible UI flows. Sign-up, sign-in, onboarding, primary value surface, payment. These are e2e candidates.

For each critical flow, classify the best layer to test it at (don't e2e a pure-function pricing calculator; don't unit-test a multi-step sign-up).

Phase 3 — Inventory existing tests

For every test file found in Phase 1:

• What it asserts — read 5–20 lines around each test(/it(/describe( block and classify:
  • Behavior test (asserts on observable outcome under varied inputs)
  • Shape test (asserts on type/structure only — usually low value)
  • Happy-path-only (no error/edge cases)
  • Snapshot only (often brittle, low signal)
• Which layer — unit (no IO, mocked deps), integration (real DB/queue/ HTTP), e2e (browser-driven, Playwright), smoke (post-deploy minimal).
• Health signals — .skip, .only, TODO comments, setTimeout waits, flaky-test workarounds, excessive mocking.

  rg -n '\.skip\(|\.only\(|@skip|setTimeout|sleep\(' tests/ test/ e2e/ src/ 2>/dev/null | rg -i 'test|spec'
  

Produce a small table:

Layer        Count  Health
Unit         12     6 behavior, 4 shape, 2 skipped
Integration  2      both behavior, both touch real DB
E2E          1      happy-path signup only
Smoke        0      —

Phase 4 — Layer-by-layer gap analysis

For each layer, judge shape of the pyramid and specific gaps.

Unit layer

• Are pure functions with branching covered? (money math, parsers, validators, scorers, classifiers, prompt builders.)

  # Pure-function candidates
  rg -nl 'export\s+(async\s+)?function\s+\w+\(' src/lib src/utils src/agents 2>/dev/null
  

• For each non-trivial function, is there a test file colocated or in __tests__/?
• Flag: branching logic with no test of either branch. Don't flag: one-line wrappers, type-only modules, re-exports.

Integration layer

• For each external boundary (DB, queue, LLM, email provider, HTTP source), is there at least one test that exercises it end-to-end within the process? (Not full e2e — just "this function actually inserts a row.")
• Common gaps: jobs that compose multiple modules (src/jobs/*.ts), source adapters (src/sources/*.ts), auth flows (token issue → consume).
• Flag: module compositions that only exist in production code paths. These are exactly where bugs hide that unit tests can't catch.

E2E layer

• For each top-of-funnel user flow (sign-up, sign-in, primary value action, payment), is there a Playwright spec?
• Flag: missing e2e for sign-up/sign-in or for the product's primary user-visible flow. Don't over-flag: every UI page does not need e2e.
• Flag in reverse: if e2e count >> integration count, the pyramid is inverted — slow, flaky CI ahead.

Smoke layer

• Are there post-deploy checks? (Health endpoint, auth round-trip, one read-only query.) If deploy is automated, smoke is non-optional.
• Flag: no smoke pack at all if deploys are automated.

Phase 5 — Quality of existing tests

This phase produces findings just like Phase 4. A green suite that doesn't actually exercise behavior is at least as dangerous as a missing suite — the missing test gets fixed when someone notices, the false- confidence test stays green forever. Read the test bodies, not just the counts. Open every test file flagged in Phase 3 and read what it actually asserts; the grep recipes below seed the investigation, they don't replace reading.

Each item below maps to a finding when present:

• Shape-only assertions. expect(result).toHaveProperty('id') on a typed return is type-system noise. Same with expect(arr).toHaveLength(3) with no follow-up on what's in the array.

  rg -n 'toHaveProperty|toBeDefined|toBeTruthy\s*\(\s*\)|toHaveLength' tests/ test/ src/ 2>/dev/null
  

• Tautological mocks. The test mocks a dependency to return X, then asserts the function returned X. This proves the mock works, not the function. Read the assertion: does it follow from the input, or from the stub? If the latter → finding.

  # Heuristic: any file with vi.mock + a matching mockResolvedValue +
  # an assertion shaped like expect(result).toEqual(<same value>).
  rg -nB2 -A2 'mockResolvedValue|mockReturnValue' tests/ test/ src/ 2>/dev/null | head -80
  

• Over-mocking. Every dependency stubbed = the test proves the stubs return what they were told to. Especially common in integration tests that should hit real DB/queue but mock them away. Count mocks per test file; >5 in a single unit test usually means refactor-not-test.
• Snapshots over rendered HTML. Large snapshots that no one reviews; whole-component HTML dumps that change on every refactor → reviewer rubber-stamps the diff.

  find . -name '*.snap' -not -path '*/node_modules/*' -exec wc -l {} +
  

• Skipped or focused tests left in. .skip/.only/xit/xdescribe/ fit/fdescribe in committed code is always a finding — a .only silently hides every other test in the file from CI.

  rg -n '\b(it|test|describe)\.(skip|only)\b|^\s*(xit|fit|xdescribe|fdescribe)\b' tests/ test/ src/ 2>/dev/null
  

• Time-dependent flakiness. new Date(), Date.now(), real timers without vi.useFakeTimers(). CI clocks drift; tests that pass locally fail at midnight UTC.

  rg -n 'new Date\(\)|Date\.now\(\)|setTimeout\(.*\d+\)' tests/ test/ src/ 2>/dev/null | rg -i 'test|spec'
  

• Real network in unit tests. Unstubbed fetch / axios / HTTP client → slow, flaky, and a CI outage on every vendor blip.

  rg -n 'fetch\(|axios|got\(|http\.get' tests/ test/ src/ 2>/dev/null | rg 'test|spec'
  

• Brittle selectors in e2e. Locators tied to deeply-nested CSS classes, generated IDs, or text that changes for i18n. Prefer roles + accessible names. Flag chained > selectors and nth-child.

  rg -n 'locator\(.*>.*\)|nth-child|nth-of-type' tests/e2e tests/playwright 2>/dev/null
  

• Happy-path only on critical logic. A function with branching has one test that hits the trivial branch. Open the test and the production code side-by-side: are the error branches covered? If the production code has try/catch, if (!x) throw, retry loops, fallback paths → are they exercised? If not → finding.
• Test name vs. test body mismatch. Test is named "rejects expired tokens" but only checks the happy path because the await expect(...) .rejects was forgotten. These pass for the wrong reason. When scanning, read every test(name, ...) and ask "would this fail if the production code were broken?"
• Tests that never assert. A test body that runs setup + invocation but no expect (or only expect(true).toBe(true)-shaped) — the test passes if the function doesn't throw, regardless of behavior.

  # Cheap heuristic: list test files with zero `expect(`. Manually
  # inspect to rule out helpers that re-export.
  for f in $(find src tests -name '*.test.*' -o -name '*.spec.*' 2>/dev/null); do
    [ "$(rg -c '\bexpect\(' "$f" 2>/dev/null || echo 0)" = "0" ] && echo "NO ASSERTIONS: $f"
  done
  

• Coverage of trivial / re-exported code, none on real logic. A 100% line-coverage report on a file that's all export { foo } from './foo' is meaningless. When a coverage report exists, sanity-check which lines it counts.

Phase 6 — CI integration

• Are tests wired into CI? Pre-merge or post-merge?
• Is e2e in a separate job so unit-test feedback stays fast?
• Is there a smoke job that runs against the deployed environment?
• Coverage report generated? (Optional — coverage is a map, not a goal.)

Report format

Emit one markdown report. Structure:

# Test Coverage Audit — <project> (<YYYY-MM-DD>)

## Executive summary

- Pyramid shape: <Healthy | Anemic | Missing-middle | Inverted | Empty>
- N findings: X Critical, Y High, Z Medium, W Low, V Info
  (G gaps · D defects in existing tests)
- Top three risks (one line each — can mix gaps and defects)
- Overall verdict (1–2 sentences)

## Pyramid snapshot

| Layer       | Count | Health | Notes                              |
| ----------- | ----- | ------ | ---------------------------------- |
| Unit        | …     | …      | …behavior / …shape-only / …skipped |
| Integration | …     | …      | …                                  |
| E2E         | …     | …      | …                                  |
| Smoke       | …     | …      | …                                  |

"Health" is `Strong | Adequate | Anemic | Compromised | Empty`. **Compromised**
is the trap to call out — count is high but the tests don't actually
exercise behavior.

## Scope & methodology

- What was reviewed (paths, commit SHA)
- What was NOT reviewed (runtime behavior, prod telemetry)
- Tools/commands used

## Critical flows considered

- List of flows identified in Phase 2 with the layer chosen for each.

## Findings (ranked, Critical → Info)

Use stable IDs (`F-001`, `F-002` …) and tag each finding's _kind_ so the
reader can scan at a glance:

- `[Kind: Gap]` — no test covers this flow / branch.
- `[Kind: Defect]` — a test exists but doesn't actually verify the behavior
  it claims to (shape-only, tautological mock, skipped, brittle, etc.).

### F-001 — <Short title> [Severity: High] [Layer: Integration] [Kind: Gap]

**Flow at risk:** Daily digest synthesis (`src/jobs/synthesize.ts`)
**Current coverage:** None.
**Why it matters:** This job composes LLM call + DB write + Resend
dispatch. A regression in any step silently breaks the product's core
value delivery; no other test layer would catch it.

**Suggested test (sketch, not code to add now):**

- Integration test, real Postgres test DB, stub Anthropic + Resend with
  recorded fixtures.
- Cases: happy path produces N digest rows; LLM 429 → job retries;
  empty input set → job no-ops without sending email.

**Verification needed (if any):** N/A — coverage gap, not a runtime claim.

### F-002 — Auth token test mocks the verifier it's verifying [Severity: High] [Layer: Unit] [Kind: Defect]

**Test:** `src/lib/feedback-token.test.ts:42-58`
**Production code at risk:** `src/lib/feedback-token.ts:18`
**What the test does:**

> ```ts
> // 3–10 line excerpt of the offending test
> ```
>
> The test mocks `verifyFeedbackToken` to return `true`, then asserts that
> the flow accepts the token — proving the mock works, not the verifier.
> A real regression in the HMAC check would not surface.

**Why it matters:** This test currently sits in the "covered" column on
any line-coverage report; the audit's executive summary would otherwise
record this flow as protected. It is not.

**Suggested fix:** Drop the mock; sign a token with the real
`signFeedbackToken` in the test and pass it through. Assert behavior
under tampered input as a separate case.

### F-003 — …

Order findings strictly by severity, then by likelihood of regression (complexity × change frequency). Use stable IDs so they can be referenced in follow-up PRs.

If the pyramid is healthy in a particular layer, say so explicitly — absence of findings in a section is itself useful signal. Same for existing-test defects: "Sampled N of M unit tests; assertions substantive, no .skip/.only, mocks limited to external boundaries" is a valuable report line.

Anti-patterns to avoid in the report

• ❌ "Coverage is at 42%, target 80%." → ✅ "The synthesis job's retry branch (src/jobs/synthesize.ts:88-104) has no test; a regression here silently drops daily emails."
• ❌ Recommending tests for trivial getters or type re-exports.
• ❌ Recommending e2e tests for pure logic that belongs in a unit test.
• ❌ Recommending unit tests for a flow that genuinely needs the DB to prove anything — that's an integration test.
• ❌ Marking everything Critical to look thorough. Follow the rubric — a missing test on a one-line utility is not Critical.
• ❌ Treating shape-only assertions as coverage. They satisfy the type system; they don't prove behavior.
• ❌ Counting test files without reading them. "10 unit test files" is not a signal; "10 unit test files, 4 contain only shape assertions" is.
• ❌ Skipping the existing-test audit because the suite is green. Green ≠ behavior-verifying. Phase 5 is mandatory, not optional.