| name | test-coverage |
| description | Perform a read-only audit of the codebase's test pyramid. Two halves of equal weight: (1) find critical flows lacking coverage; (2) audit *existing* tests for false confidence — shape-only assertions, tautological mocks, skipped/.only blocks, brittle selectors, tests that pass for the wrong reason. A green suite that doesn't exercise behavior is at least as dangerous as a missing one. Assumes Vitest for unit/integration and Playwright for e2e/smoke. Pragmatic by contract: does NOT chase 100% line coverage — focuses on logic, branching, money math, auth/tenant filters, and user-visible flows. Produces one ranked report with severity, finding kind (Gap vs Defect), evidence (file:line), and a concrete suggested fix. Use whenever the user asks for a "test audit", "test coverage review", "test pyramid health check", "audit our tests", "what should we test", or similar. Never modifies test code — read-only by contract.
|
| allowed-tools | Bash(rg:*), Bash(grep:*), Bash(find:*), Bash(ls:*), Bash(wc:*), Bash(cat:*), Bash(sort:*), Bash(uniq:*), Bash(head:*), Bash(tail:*), Bash(awk:*), Bash(sed:*), Bash(jq:*), Bash(node:*), Bash(npm:*), Bash(pnpm:*), Bash(git:*), Read, Write |
Test Coverage Audit
A read-only audit of the testing pyramid. The skill is methodology, not
magic: walk the phases below, gather concrete file:line evidence of what
is and is not tested, then emit one ranked report.
Operating contract
- Read-only. Do not create or modify test files, fixtures, or config.
- Evidence-first. Every finding cites
path/to/file.ts:LINE for the
production code at risk and (when relevant) the existing test that does
or doesn't cover it. No vague claims like "tests should be expanded."
- Audit existing tests, not just gaps. Half the skill's job is finding
missing coverage; the other half is judging existing tests — shape-
only assertions, over-mocking, snapshot rot, skipped/
.only blocks, real
network in unit tests, brittle e2e selectors, tests that pass for the
wrong reason. A green suite that doesn't actually test behavior is worse
than no suite, because it manufactures false confidence. Phase 5 is not
optional — it's a peer of Phase 4.
- Pragmatic, not exhaustive. The target is a healthy pyramid covering
critical flows — NOT 100% line coverage. Skip trivial code (pure type
helpers, single-line wrappers, shape-only DTOs). Focus on logic, branching,
integrations, and user-visible flows.
- Test logic, not shape. A "good test" exercises behavior under
conditions that could plausibly fail. A test that asserts
expect(obj). toHaveProperty('id') on a typed object is shape-only and should be
flagged as low-value, not counted as coverage.
The pyramid
Use this mental model when judging health. Each layer answers a different
question and has a different cost/speed/confidence tradeoff.
| Layer | Question it answers | Typical tool | Cost |
|---|
| Unit | Does this function's logic do the right thing across its branches? | Vitest (node env) | Fast, cheap |
| Integration | Do these modules + the DB/queue/HTTP boundary cooperate correctly? | Vitest + real DB/queue or testcontainers | Medium |
| E2E | Can a real user complete a critical flow end-to-end in a browser? | Playwright | Slow, expensive |
| Smoke | After deploy, is the system minimally alive? (health, auth, one happy path) | Playwright (subset) or curl | Fast, run post-deploy |
Healthy pyramid: many cheap unit tests at the base, fewer integration
tests in the middle, a small set of e2e flows for happy + key error paths,
a tiny set of smoke checks that run against production. Inverted pyramids
(e2e-heavy) are slow and flaky; missing-middle pyramids (unit + e2e only)
miss boundary bugs.
Severity rubric for findings
Findings come in two flavors: gaps (missing tests) and defects in
existing tests (false confidence). Both use the same scale.
| Severity | Gap example | Defect-in-existing example |
|---|
| Critical | A core revenue/data/security flow has no test of any kind. | An existing test for a critical flow asserts only shape / mocks every dependency / is permanently .skip'd — it appears green but verifies nothing. |
| High | Complex business logic has only happy-path coverage; error paths untested. | A test passes because it asserts against the same mock it set up (tautological); or covers only the happy path while claiming to cover the function. |
| Medium | An important flow is tested at the wrong layer (complex logic only via slow flaky e2e). | Stale snapshot covering rendered HTML that changes on every refactor; test uses Date.now() / real timers and is flaky in CI; test reaches live external services. |
| Low | Missing tests on non-critical helpers; duplication that increases maintenance cost. | Brittle CSS-selector-based e2e; over-broad mocking that hides regressions; redundant tests asserting the same property at multiple layers. |
| Info | Hygiene observation: configuration suggestions, no-runner-found. | A single stray .only left in; missing test naming convention; tests in the wrong directory. |
A finding's severity reflects the cost of a regression slipping through,
not the size of the fix. Bias toward the higher of likelihood and impact
when they disagree. A passing test that doesn't actually exercise its
target is at least as dangerous as a missing one — the missing test gets
fixed when someone notices; the false-confidence test stays green forever.
Audit phases
Work these in order. Don't skip phases even if early ones look clean —
different gaps surface in different phases.
Phase 1 — Recon
Build a map of what exists before judging what's missing.
find src -type d | head -60
cat package.json | jq '.scripts, .dependencies, .devDependencies'
cat package.json | jq '.scripts | to_entries | map(select(.value | test("vitest|playwright|jest|mocha"))) | .[]'
ls vitest.config* playwright.config* jest.config* 2>/dev/null
find . -type f \( -name '*.test.ts' -o -name '*.test.tsx' -o -name '*.spec.ts' -o -name '*.spec.tsx' -o -name '*.e2e.ts' \) -not -path '*/node_modules/*'
find . -type d \( -name '__tests__' -o -name 'tests' -o -name 'test' -o -name 'e2e' -o -name 'smoke' \) -not -path '*/node_modules/*'
cat .github/workflows/*.yml 2>/dev/null | rg -i 'test|vitest|playwright' -A2 -B1
Note: production framework, ORM, queue/worker, auth library, external
integrations (LLM, email, payment, scraping). These are the boundaries where
integration tests live.
Phase 2 — Map production surfaces to "what should be tested"
You can't judge missing tests without knowing what flows matter. Build a
shortlist of critical flows by inspecting:
- Routes / endpoints. Each is a flow entry point.
find src/routes src/app src/pages -type f 2>/dev/null
rg -n 'createServerFn|defineHandler|export\s+(async\s+)?function\s+(GET|POST|PUT|DELETE|PATCH)' src/
- Jobs / workers. Each scheduled or queued job is a critical flow.
find src/jobs src/worker -type f 2>/dev/null
rg -n 'boss\.send|boss\.work|cron|schedule' src/
- Money/auth/data-isolation code paths. Pricing, billing, auth, tenant
filters — high-cost-of-regression by definition.
rg -n 'price|amount|cents|charge|invoice|stripe' src/
rg -n 'auth|session|userId|orgId|tenantId' src/ | head -40
- External integrations. LLM/email/scraping adapters — boundary code
that benefits most from integration tests.
- User-visible UI flows. Sign-up, sign-in, onboarding, primary value
surface, payment. These are e2e candidates.
For each critical flow, classify the best layer to test it at (don't
e2e a pure-function pricing calculator; don't unit-test a multi-step
sign-up).
Phase 3 — Inventory existing tests
For every test file found in Phase 1:
Produce a small table:
Layer Count Health
Unit 12 6 behavior, 4 shape, 2 skipped
Integration 2 both behavior, both touch real DB
E2E 1 happy-path signup only
Smoke 0 —
Phase 4 — Layer-by-layer gap analysis
For each layer, judge shape of the pyramid and specific gaps.
Unit layer
Integration layer
- For each external boundary (DB, queue, LLM, email provider, HTTP source),
is there at least one test that exercises it end-to-end within the
process? (Not full e2e — just "this function actually inserts a row.")
- Common gaps: jobs that compose multiple modules (
src/jobs/*.ts),
source adapters (src/sources/*.ts), auth flows (token issue → consume).
- Flag: module compositions that only exist in production code paths.
These are exactly where bugs hide that unit tests can't catch.
E2E layer
- For each top-of-funnel user flow (sign-up, sign-in, primary value action,
payment), is there a Playwright spec?
- Flag: missing e2e for sign-up/sign-in or for the product's primary
user-visible flow. Don't over-flag: every UI page does not need e2e.
- Flag in reverse: if e2e count >> integration count, the pyramid is
inverted — slow, flaky CI ahead.
Smoke layer
- Are there post-deploy checks? (Health endpoint, auth round-trip, one
read-only query.) If deploy is automated, smoke is non-optional.
- Flag: no smoke pack at all if deploys are automated.
Phase 5 — Quality of existing tests
This phase produces findings just like Phase 4. A green suite that
doesn't actually exercise behavior is at least as dangerous as a missing
suite — the missing test gets fixed when someone notices, the false-
confidence test stays green forever. Read the test bodies, not just the
counts. Open every test file flagged in Phase 3 and read what it
actually asserts; the grep recipes below seed the investigation, they
don't replace reading.
Each item below maps to a finding when present:
- Shape-only assertions.
expect(result).toHaveProperty('id') on a
typed return is type-system noise. Same with expect(arr).toHaveLength(3)
with no follow-up on what's in the array.
rg -n 'toHaveProperty|toBeDefined|toBeTruthy\s*\(\s*\)|toHaveLength' tests/ test/ src/ 2>/dev/null
- Tautological mocks. The test mocks a dependency to return X, then
asserts the function returned X. This proves the mock works, not the
function. Read the assertion: does it follow from the input, or from
the stub? If the latter → finding.
rg -nB2 -A2 'mockResolvedValue|mockReturnValue' tests/ test/ src/ 2>/dev/null | head -80
- Over-mocking. Every dependency stubbed = the test proves the stubs
return what they were told to. Especially common in integration tests
that should hit real DB/queue but mock them away. Count mocks per test
file; >5 in a single unit test usually means refactor-not-test.
- Snapshots over rendered HTML. Large snapshots that no one reviews;
whole-component HTML dumps that change on every refactor → reviewer
rubber-stamps the diff.
find . -name '*.snap' -not -path '*/node_modules/*' -exec wc -l {} +
- Skipped or focused tests left in.
.skip/.only/xit/xdescribe/
fit/fdescribe in committed code is always a finding — a .only
silently hides every other test in the file from CI.
rg -n '\b(it|test|describe)\.(skip|only)\b|^\s*(xit|fit|xdescribe|fdescribe)\b' tests/ test/ src/ 2>/dev/null
- Time-dependent flakiness.
new Date(), Date.now(), real timers
without vi.useFakeTimers(). CI clocks drift; tests that pass locally
fail at midnight UTC.
rg -n 'new Date\(\)|Date\.now\(\)|setTimeout\(.*\d+\)' tests/ test/ src/ 2>/dev/null | rg -i 'test|spec'
- Real network in unit tests. Unstubbed
fetch / axios / HTTP
client → slow, flaky, and a CI outage on every vendor blip.
rg -n 'fetch\(|axios|got\(|http\.get' tests/ test/ src/ 2>/dev/null | rg 'test|spec'
- Brittle selectors in e2e. Locators tied to deeply-nested CSS
classes, generated IDs, or text that changes for i18n. Prefer roles +
accessible names. Flag chained
> selectors and nth-child.
rg -n 'locator\(.*>.*\)|nth-child|nth-of-type' tests/e2e tests/playwright 2>/dev/null
- Happy-path only on critical logic. A function with branching has
one test that hits the trivial branch. Open the test and the
production code side-by-side: are the error branches covered? If the
production code has try/catch,
if (!x) throw, retry loops, fallback
paths → are they exercised? If not → finding.
- Test name vs. test body mismatch. Test is named "rejects expired
tokens" but only checks the happy path because the
await expect(...) .rejects was forgotten. These pass for the wrong reason. When
scanning, read every test(name, ...) and ask "would this fail if
the production code were broken?"
- Tests that never assert. A test body that runs setup + invocation
but no
expect (or only expect(true).toBe(true)-shaped) — the test
passes if the function doesn't throw, regardless of behavior.
for f in $(find src tests -name '*.test.*' -o -name '*.spec.*' 2>/dev/null); do
[ "$(rg -c '\bexpect\(' "$f" 2>/dev/null || echo 0)" = "0" ] && echo "NO ASSERTIONS: $f"
done
- Coverage of trivial / re-exported code, none on real logic. A
100% line-coverage report on a file that's all
export { foo } from './foo' is meaningless. When a coverage report exists, sanity-check
which lines it counts.
Phase 6 — CI integration
- Are tests wired into CI? Pre-merge or post-merge?
- Is e2e in a separate job so unit-test feedback stays fast?
- Is there a smoke job that runs against the deployed environment?
- Coverage report generated? (Optional — coverage is a map, not a goal.)
Report format
Emit one markdown report. Structure:
# Test Coverage Audit — <project> (<YYYY-MM-DD>)
## Executive summary
- Pyramid shape: <Healthy | Anemic | Missing-middle | Inverted | Empty>
- N findings: X Critical, Y High, Z Medium, W Low, V Info
(G gaps · D defects in existing tests)
- Top three risks (one line each — can mix gaps and defects)
- Overall verdict (1–2 sentences)
## Pyramid snapshot
| Layer | Count | Health | Notes |
| ----------- | ----- | ------ | ---------------------------------- |
| Unit | … | … | …behavior / …shape-only / …skipped |
| Integration | … | … | … |
| E2E | … | … | … |
| Smoke | … | … | … |
"Health" is `Strong | Adequate | Anemic | Compromised | Empty`. **Compromised**
is the trap to call out — count is high but the tests don't actually
exercise behavior.
## Scope & methodology
- What was reviewed (paths, commit SHA)
- What was NOT reviewed (runtime behavior, prod telemetry)
- Tools/commands used
## Critical flows considered
- List of flows identified in Phase 2 with the layer chosen for each.
## Findings (ranked, Critical → Info)
Use stable IDs (`F-001`, `F-002` …) and tag each finding's _kind_ so the
reader can scan at a glance:
- `[Kind: Gap]` — no test covers this flow / branch.
- `[Kind: Defect]` — a test exists but doesn't actually verify the behavior
it claims to (shape-only, tautological mock, skipped, brittle, etc.).
### F-001 — <Short title> [Severity: High] [Layer: Integration] [Kind: Gap]
**Flow at risk:** Daily digest synthesis (`src/jobs/synthesize.ts`)
**Current coverage:** None.
**Why it matters:** This job composes LLM call + DB write + Resend
dispatch. A regression in any step silently breaks the product's core
value delivery; no other test layer would catch it.
**Suggested test (sketch, not code to add now):**
- Integration test, real Postgres test DB, stub Anthropic + Resend with
recorded fixtures.
- Cases: happy path produces N digest rows; LLM 429 → job retries;
empty input set → job no-ops without sending email.
**Verification needed (if any):** N/A — coverage gap, not a runtime claim.
### F-002 — Auth token test mocks the verifier it's verifying [Severity: High] [Layer: Unit] [Kind: Defect]
**Test:** `src/lib/feedback-token.test.ts:42-58`
**Production code at risk:** `src/lib/feedback-token.ts:18`
**What the test does:**
> ```ts
> // 3–10 line excerpt of the offending test
> ```
>
> The test mocks `verifyFeedbackToken` to return `true`, then asserts that
> the flow accepts the token — proving the mock works, not the verifier.
> A real regression in the HMAC check would not surface.
**Why it matters:** This test currently sits in the "covered" column on
any line-coverage report; the audit's executive summary would otherwise
record this flow as protected. It is not.
**Suggested fix:** Drop the mock; sign a token with the real
`signFeedbackToken` in the test and pass it through. Assert behavior
under tampered input as a separate case.
### F-003 — …
Order findings strictly by severity, then by likelihood of regression
(complexity × change frequency). Use stable IDs so they can be referenced
in follow-up PRs.
If the pyramid is healthy in a particular layer, say so explicitly —
absence of findings in a section is itself useful signal. Same for
existing-test defects: "Sampled N of M unit tests; assertions
substantive, no .skip/.only, mocks limited to external boundaries"
is a valuable report line.
Anti-patterns to avoid in the report
- ❌ "Coverage is at 42%, target 80%." → ✅ "The synthesis job's retry
branch (
src/jobs/synthesize.ts:88-104) has no test; a regression here
silently drops daily emails."
- ❌ Recommending tests for trivial getters or type re-exports.
- ❌ Recommending e2e tests for pure logic that belongs in a unit test.
- ❌ Recommending unit tests for a flow that genuinely needs the DB to
prove anything — that's an integration test.
- ❌ Marking everything Critical to look thorough. Follow the rubric — a
missing test on a one-line utility is not Critical.
- ❌ Treating shape-only assertions as coverage. They satisfy the type
system; they don't prove behavior.
- ❌ Counting test files without reading them. "10 unit test files" is
not a signal; "10 unit test files, 4 contain only shape assertions" is.
- ❌ Skipping the existing-test audit because the suite is green. Green
≠ behavior-verifying. Phase 5 is mandatory, not optional.
