test-audit

name	test-audit
description	Static test-suite quality audit - coverage from lcov/Istanbul/Cobertura/go/tarpaulin reports, pyramid shape (unit/integration/e2e ratio), anti-patterns (.only leaks, skipped tests, no-assertion tests, hardcoded sleeps). Stack-aware across 11 supported stacks.
user-invocable	true
model	sonnet
context	fork
argument-hint	["target:path:<dir>|target:file:<glob>|target:coverage:<path>|mode:all"]
allowed-tools	Read Glob Grep Bash

Scope for v1

• Static analysis only. Parses coverage reports produced by the project's test runner. Does not execute tests, does not re-run the suite, does not query CI history.
• Flaky-test detection is deferred. Detecting non-deterministic failures requires multiple runs or CI-history access, both out of scope for the first ship. Tracked as future-work item TEST-flake-detection.
• Live flake / performance probes: revisit after real usage.

---

Configuration (adapt before first run)

Replace these placeholders:

• [TEST_PATH] - primary location of test files (e.g. src/**/*.test.ts, tests/, spec/, Tests/, src/test/)
• [COVERAGE_PATH] - location of coverage reports if non-standard (e.g. coverage/lcov.info, reports/coverage.xml)
• [APP_SOURCE_PATH] - path to application source (for colocated-test detection) - e.g. src/, app/, lib/

---

Step 0 - Target resolution

Parse $ARGUMENTS for a target: or mode: token.

Pattern	Meaning
target:path:<dir>	Audit tests under a specific directory only (e.g. target:path:src/auth)
target:file:<glob>	Audit files matching the glob (e.g. target:file:**/*.integration.test.ts)
target:coverage:<path>	Force a specific coverage report path instead of auto-detection
mode:all / no argument	Full audit - every discovered test file and auto-detected coverage report.

STRICT PARSING: derive target ONLY from explicit text in $ARGUMENTS. Do NOT infer from conversation context, recent blocks, or memory.

Announce: Running test-audit - scope: [FULL | target: <resolved>] - stack: <pending detection>

---

Step 1 - Stack detection

Detect the primary stack in this priority order. First matching marker wins.

Stack	Marker file(s)
node-ts	tsconfig.json + package.json
node-js	package.json (no tsconfig.json)
python	pyproject.toml or requirements.txt or setup.py
go	go.mod
rust	Cargo.toml
swift	Package.swift or *.xcodeproj / *.xcworkspace
kotlin	build.gradle.kts or build.gradle with kotlin( plugin
dotnet	*.csproj or *.sln
ruby	Gemfile
java	pom.xml or build.gradle (no Kotlin plugin)
generic	None of the above

If stack is generic: run T1-T8 with generic patterns only, skip framework-specific coverage parsing, announce No known stack detected - running generic test-quality checks only.

Update announcement: Running test-audit - scope: [...] - stack: <detected>

---

Step 2 - Framework detection

Within the detected stack, identify the test framework. Parses package.json scripts, config files, and dependency declarations.

Stack	Framework detection
node-ts / node-js	(a) vitest in devDependencies or vitest.config.* present -> vitest (b) jest in devDeps or jest.config.* -> jest (c) mocha in devDeps or .mocharc* -> mocha (d) "test": "node --test" in scripts -> node:test (e) none -> unknown
python	(a) pytest in [tool.pytest.ini_options] of pyproject.toml or pytest.ini -> pytest (b) import unittest in any test file -> unittest (c) none -> unknown
go	Always testing (stdlib). Presence of testify in go.mod noted.
rust	Always cargo test. Tarpaulin presence in dev-dependencies noted.
swift	(a) import Testing in any test file -> swift-testing (b) import XCTest -> XCTest (c) both -> hybrid
kotlin	(a) kotest- in dependencies -> Kotest (b) junit -> JUnit (c) none -> unknown
dotnet	(a) xunit NuGet package -> xUnit (b) NUnit -> NUnit (c) MSTest.TestFramework -> MSTest (d) none -> unknown
ruby	(a) rspec in Gemfile -> RSpec (b) minitest -> Minitest (c) none -> unknown
java	(a) junit-jupiter in pom.xml or build.gradle -> JUnit 5 (b) junit 4.x -> JUnit 4 (c) testng -> TestNG (d) none -> unknown

Record: Framework: <name> - used to select the correct grep patterns in Step 6.

---

Step 3 - Test file discovery

Glob by stack convention. Include both co-located and directory-based tests.

Stack	Glob patterns
node-ts / node-js	**/*.test.{ts,tsx,js,jsx,mjs}, **/*.spec.{ts,tsx,js,jsx,mjs}, **/__tests__/**/*.{ts,tsx,js,jsx,mjs}, tests/**/*.{ts,tsx,js,jsx,mjs}
python	test_*.py, *_test.py, tests/**/*.py
go	**/*_test.go
rust	tests/**/*.rs, any src/**/*.rs with #[cfg(test)] module
swift	Tests/**/*.swift, *Tests/**/*.swift
kotlin	src/test/**/*.kt, src/androidTest/**/*.kt
dotnet	**/*Tests*.cs, **/*Test.cs, under any *.Tests.csproj
ruby	spec/**/*_spec.rb, test/**/*_test.rb
java	src/test/**/*.java

Respect target:path: / target:file: if provided.

Announce: Discovered N test files across K layers.

If N == 0: output No test files found. Skipping static checks. Coverage reports still parsed if available. and continue to Step 4.

---

Step 4 - Coverage report parsing (C1-C3)

Auto-detect coverage report formats. Honor target:coverage:<path> if provided.

C1 - lcov.info (node, rust with tarpaulin --out Lcov)

Search paths: coverage/lcov.info, coverage/lcov-report/, lcov.info, reports/lcov.info.

Parse by scanning lines:

• SF:<file> starts a file block.
• LF:<n> = lines found, LH:<n> = lines hit. File coverage = LH / LF * 100.
• BRF:<n> = branches found, BRH:<n> = branches hit.
• end_of_record terminates the file block.

Report: total line %, total branch %, files < 50% (top 10 by SLOC), files at 0% (top 10).

C2 - Istanbul JSON (node)

Search paths: coverage/coverage-summary.json, coverage/coverage-final.json.

Parse JSON. For coverage-summary.json: read the total key (lines.pct, branches.pct, functions.pct, statements.pct). For coverage-final.json: aggregate per-file s (statements) / b (branches) / f (functions).

Report: line %, branch %, function %, statement %.

C3 - Cobertura XML (python, java, dotnet)

Search paths: coverage.xml, cobertura.xml, coverage/cobertura-coverage.xml, build/reports/cobertura/coverage.xml.

Parse XML: <coverage line-rate="0.873" branch-rate="0.712"> at root. Per-package and per-class rates in child elements.

Report: line %, branch %, top uncovered classes.

C3 (go) - coverage.out

Search paths: coverage.out, cover.out, .coverage/coverage.out.

Parse format: mode: <mode>\n<file>:<start>,<end> <stmts> <count>. Total % = stmts_hit / stmts_total. Use Bash with go tool cover -func=<file> if go is on PATH for a cleaner summary.

C3 (rust) - tarpaulin JSON

Search paths: tarpaulin-report.json, target/tarpaulin/report.json.

Parse JSON: files[].coverage percentages, aggregate weighted by file SLOC.

C3 (swift) - xcresult bundle (optional)

If *.xcresult directory exists AND xcrun is on PATH: run xcrun xccov view --report --json <file>.xcresult and parse JSON output. Skip gracefully if xcrun is unavailable (CI or Linux).

Report format

## Coverage (source: <format>)
- Total lines: <X>%
- Branches: <Y>% (if available)
- Functions: <Z>% (if available)
- Files below 50%: <N>
- Files at 0%: <N>

Severity mapping

• Total < 50% line coverage -> High
• Total < 80% line coverage -> Medium
• Any changed file in the current block at 0% coverage -> Critical (requires git diff cross-check; if git history unavailable, report as High)
• No coverage report detected -> nudge: No coverage report detected. Pass target:coverage:<path> or run <test-command> with coverage flag to generate one. - no severity, just a note.

---

Step 5 - Pyramid shape (P1-P3)

Classify each test file discovered in Step 3:

ID	Layer	Path / content heuristic
P1	Unit	Colocated *.test.* / *.spec.* next to source, or under tests/unit/**, __tests__/**, test/unit/**, spec/unit/**, src/test/**/unit/**. Default bucket if nothing else matches.
P2	Integration	Path contains integration/, or filename matches *.integration.test.* / *.integration.spec.* / *_integration_test.*.
P3	E2E	Path contains e2e/, end-to-end/, uat/, system/, or filename matches *.e2e.*. File imports @playwright/test, cypress, detox, selenium, webdriverio, appium, or (python) playwright, splinter.

Compute N_unit / N_integration / N_e2e and their percentages of total.

Verdicts (target pyramid: ~70/20/10):

Condition	Severity	Note
N_e2e > 30%	Medium	Inverted pyramid - E2E-heavy suite is slow and brittle
N_integration > 50%	Medium	Middle-heavy - consider splitting into unit vs E2E
N_unit == 0 AND total > 0	High	No unit tests - every change goes through slow layers
Only one layer present	Medium	Narrow coverage - regressions in untested layers will slip through
Within 50-85% unit / 10-35% integration / 2-20% E2E	OK	Report as balanced

Report:

## Pyramid
- Unit: <N> (<P>%)
- Integration: <N> (<P>%)
- E2E: <N> (<P>%)
- Verdict: [balanced | inverted | middle-heavy | narrow | unit-absent]

---

Step 6 - Static anti-pattern checks (T1-T8)

Run per stack. Read ${CLAUDE_SKILL_DIR}/PATTERNS.md for stack-specific grep patterns for all checks below. Checks without a matching pattern produce N/A - skipped for <stack>, not false positives.

T1 - .only / fit / fdescribe committed

Critical. Focused tests in committed code break CI: suite passes locally but skips everything except the focused case. Grep for focused-test markers using patterns from PATTERNS.md → T1.

Report each match with FILE:LINE. If ≥ 2 files have matches, escalate to T8.

T2 - Skipped tests

Medium per match; High if skipped tests > 10% of suite. Grep for skip markers using patterns from PATTERNS.md → T2.

Report: count and top 10 with file:line.

T3 - .todo placeholders

Low. Placeholder tests that never fail. Flag when count grows; not actionable per instance. See PATTERNS.md → T3.

Report: count only.

T4 - Empty test bodies

High. Empty tests always pass and create false confidence. Grep for empty test body patterns from PATTERNS.md → T4 (multiline regex - use multiline: true where supported).

Report: each match with file:line.

T5 - Tests without assertions

High. Test that runs code paths without verifying outcomes.

Heuristic: within a detected test body, grep for at least one of the stack's assertion patterns from PATTERNS.md → T5. Flag any test body that lacks all of its framework's patterns.

Report: each match with file:line. Expect some false positives (helper-only tests, integration setup) - mark findings as probable when the body is very short or wraps another function.

T6 - Hardcoded sleeps in tests

Medium. Sleep-based waits are the primary source of flaky tests. Grep for sleep/delay patterns (numeric sleep ≥ 500ms) using PATTERNS.md → T6.

Report: each match with file:line and duration.

T7 - Debug output left in tests

Low. Doesn't affect correctness but adds noise. Grep test files for debug output patterns from PATTERNS.md → T7.

Report: count only. Do not list individually unless count > 20.

T8 - Multiple .only across files (broken CI)

Critical. Derived from T1: if ≥ 2 distinct files contain T1 matches, the CI gate is broken in a non-obvious way (each file contributes a focused case, so several cases run but most are skipped).

Report: one aggregate finding with the list of affected files.

---

Step 7 - Report and backlog decision gate

Generate the report using the template in ${CLAUDE_SKILL_DIR}/REPORT.md. Apply the severity guide and backlog writing rules from the same file.

---

Execution notes

• Do NOT modify any test file. Audit only.
• Do NOT execute tests or the test runner. Coverage reports must already exist on disk.
• target:coverage:<path> bypasses auto-detection; accept absolute or project-relative path.
• If git history is available, cross-reference C Critical findings against git diff HEAD~1 (files changed in the most recent commit) to distinguish "0% on pre-existing file" (High) from "0% on file changed in this block" (Critical). If git is not available, downgrade all 0% findings to High.
• This skill complements the Phase 3 test execution (which runs [TEST_COMMAND] but never inspects the suite itself). Run in Phase 5d Track C after Phase 3 passes.
• After the report, ask: "Do you want me to prepare the corrections for the identified findings?" Reply with yes only after the user has signed off on the specific findings.