| name | test-audit |
| description | Static test-suite quality audit - coverage from lcov/Istanbul/Cobertura/go/tarpaulin reports, pyramid shape (unit/integration/e2e ratio), anti-patterns (.only leaks, skipped tests, no-assertion tests, hardcoded sleeps). Stack-aware across 11 supported stacks. |
| user-invocable | true |
| model | sonnet |
| context | fork |
| argument-hint | ["target:path:<dir>|target:file:<glob>|target:coverage:<path>|mode:all"] |
| allowed-tools | Read Glob Grep Bash |
Scope for v1
- Static analysis only. Parses coverage reports produced by the project's test runner. Does not execute tests, does not re-run the suite, does not query CI history.
- Flaky-test detection is deferred. Detecting non-deterministic failures requires multiple runs or CI-history access, both out of scope for the first ship. Tracked as future-work item
TEST-flake-detection.
- Live flake / performance probes: revisit after real usage.
Configuration (adapt before first run)
Replace these placeholders:
[TEST_PATH] - primary location of test files (e.g. src/**/*.test.ts, tests/, spec/, Tests/, src/test/)[COVERAGE_PATH] - location of coverage reports if non-standard (e.g. coverage/lcov.info, reports/coverage.xml)[APP_SOURCE_PATH] - path to application source (for colocated-test detection) - e.g. src/, app/, lib/
Step 0 - Target resolution
Parse $ARGUMENTS for a target: or mode: token.
| Pattern | Meaning |
|---|
target:path:<dir> | Audit tests under a specific directory only (e.g. target:path:src/auth) |
target:file:<glob> | Audit files matching the glob (e.g. target:file:**/*.integration.test.ts) |
target:coverage:<path> | Force a specific coverage report path instead of auto-detection |
mode:all / no argument | Full audit - every discovered test file and auto-detected coverage report. |
STRICT PARSING: derive target ONLY from explicit text in $ARGUMENTS. Do NOT infer from conversation context, recent blocks, or memory.
Announce: Running test-audit - scope: [FULL | target: <resolved>] - stack: <pending detection>
Step 1 - Stack detection
Detect the primary stack in this priority order. First matching marker wins.
| Stack | Marker file(s) |
|---|
| node-ts | tsconfig.json + package.json |
| node-js | package.json (no tsconfig.json) |
| python | pyproject.toml or requirements.txt or setup.py |
| go | go.mod |
| rust | Cargo.toml |
| swift | Package.swift or *.xcodeproj / *.xcworkspace |
| kotlin | build.gradle.kts or build.gradle with kotlin( plugin |
| dotnet | *.csproj or *.sln |
| ruby | Gemfile |
| java | pom.xml or build.gradle (no Kotlin plugin) |
| generic | None of the above |
If stack is generic: run T1-T8 with generic patterns only, skip framework-specific coverage parsing, announce No known stack detected - running generic test-quality checks only.
Update announcement: Running test-audit - scope: [...] - stack: <detected>
Step 2 - Framework detection
Within the detected stack, identify the test framework. Parses package.json scripts, config files, and dependency declarations.
| Stack | Framework detection |
|---|
| node-ts / node-js | (a) vitest in devDependencies or vitest.config.* present -> vitest (b) jest in devDeps or jest.config.* -> jest (c) mocha in devDeps or .mocharc* -> mocha (d) "test": "node --test" in scripts -> node:test (e) none -> unknown |
| python | (a) pytest in [tool.pytest.ini_options] of pyproject.toml or pytest.ini -> pytest (b) import unittest in any test file -> unittest (c) none -> unknown |
| go | Always testing (stdlib). Presence of testify in go.mod noted. |
| rust | Always cargo test. Tarpaulin presence in dev-dependencies noted. |
| swift | (a) import Testing in any test file -> swift-testing (b) import XCTest -> XCTest (c) both -> hybrid |
| kotlin | (a) kotest- in dependencies -> Kotest (b) junit -> JUnit (c) none -> unknown |
| dotnet | (a) xunit NuGet package -> xUnit (b) NUnit -> NUnit (c) MSTest.TestFramework -> MSTest (d) none -> unknown |
| ruby | (a) rspec in Gemfile -> RSpec (b) minitest -> Minitest (c) none -> unknown |
| java | (a) junit-jupiter in pom.xml or build.gradle -> JUnit 5 (b) junit 4.x -> JUnit 4 (c) testng -> TestNG (d) none -> unknown |
Record: Framework: <name> - used to select the correct grep patterns in Step 6.
Step 3 - Test file discovery
Glob by stack convention. Include both co-located and directory-based tests.
| Stack | Glob patterns |
|---|
| node-ts / node-js | **/*.test.{ts,tsx,js,jsx,mjs}, **/*.spec.{ts,tsx,js,jsx,mjs}, **/__tests__/**/*.{ts,tsx,js,jsx,mjs}, tests/**/*.{ts,tsx,js,jsx,mjs} |
| python | test_*.py, *_test.py, tests/**/*.py |
| go | **/*_test.go |
| rust | tests/**/*.rs, any src/**/*.rs with #[cfg(test)] module |
| swift | Tests/**/*.swift, *Tests/**/*.swift |
| kotlin | src/test/**/*.kt, src/androidTest/**/*.kt |
| dotnet | **/*Tests*.cs, **/*Test.cs, under any *.Tests.csproj |
| ruby | spec/**/*_spec.rb, test/**/*_test.rb |
| java | src/test/**/*.java |
Respect target:path: / target:file: if provided.
Announce: Discovered N test files across K layers.
If N == 0: output No test files found. Skipping static checks. Coverage reports still parsed if available. and continue to Step 4.
Step 4 - Coverage report parsing (C1-C3)
Auto-detect coverage report formats. Honor target:coverage:<path> if provided.
C1 - lcov.info (node, rust with tarpaulin --out Lcov)
Search paths: coverage/lcov.info, coverage/lcov-report/, lcov.info, reports/lcov.info.
Parse by scanning lines:
SF:<file> starts a file block.LF:<n> = lines found, LH:<n> = lines hit. File coverage = LH / LF * 100.BRF:<n> = branches found, BRH:<n> = branches hit.end_of_record terminates the file block.
Report: total line %, total branch %, files < 50% (top 10 by SLOC), files at 0% (top 10).
C2 - Istanbul JSON (node)
Search paths: coverage/coverage-summary.json, coverage/coverage-final.json.
Parse JSON. For coverage-summary.json: read the total key (lines.pct, branches.pct, functions.pct, statements.pct). For coverage-final.json: aggregate per-file s (statements) / b (branches) / f (functions).
Report: line %, branch %, function %, statement %.
C3 - Cobertura XML (python, java, dotnet)
Search paths: coverage.xml, cobertura.xml, coverage/cobertura-coverage.xml, build/reports/cobertura/coverage.xml.
Parse XML: <coverage line-rate="0.873" branch-rate="0.712"> at root. Per-package and per-class rates in child elements.
Report: line %, branch %, top uncovered classes.
C3 (go) - coverage.out
Search paths: coverage.out, cover.out, .coverage/coverage.out.
Parse format: mode: <mode>\n<file>:<start>,<end> <stmts> <count>. Total % = stmts_hit / stmts_total. Use Bash with go tool cover -func=<file> if go is on PATH for a cleaner summary.
C3 (rust) - tarpaulin JSON
Search paths: tarpaulin-report.json, target/tarpaulin/report.json.
Parse JSON: files[].coverage percentages, aggregate weighted by file SLOC.
C3 (swift) - xcresult bundle (optional)
If *.xcresult directory exists AND xcrun is on PATH: run xcrun xccov view --report --json <file>.xcresult and parse JSON output. Skip gracefully if xcrun is unavailable (CI or Linux).
Report format
## Coverage (source: <format>)
- Total lines: <X>%
- Branches: <Y>% (if available)
- Functions: <Z>% (if available)
- Files below 50%: <N>
- Files at 0%: <N>
Severity mapping
- Total < 50% line coverage -> High
- Total < 80% line coverage -> Medium
- Any changed file in the current block at 0% coverage -> Critical (requires git diff cross-check; if git history unavailable, report as High)
- No coverage report detected -> nudge:
No coverage report detected. Pass target:coverage:<path> or run <test-command> with coverage flag to generate one. - no severity, just a note.
Step 5 - Pyramid shape (P1-P3)
Classify each test file discovered in Step 3:
| ID | Layer | Path / content heuristic |
|---|
| P1 | Unit | Colocated *.test.* / *.spec.* next to source, or under tests/unit/**, __tests__/**, test/unit/**, spec/unit/**, src/test/**/unit/**. Default bucket if nothing else matches. |
| P2 | Integration | Path contains integration/, or filename matches *.integration.test.* / *.integration.spec.* / *_integration_test.*. |
| P3 | E2E | Path contains e2e/, end-to-end/, uat/, system/, or filename matches *.e2e.*. File imports @playwright/test, cypress, detox, selenium, webdriverio, appium, or (python) playwright, splinter. |
Compute N_unit / N_integration / N_e2e and their percentages of total.
Verdicts (target pyramid: ~70/20/10):
| Condition | Severity | Note |
|---|
N_e2e > 30% | Medium | Inverted pyramid - E2E-heavy suite is slow and brittle |
N_integration > 50% | Medium | Middle-heavy - consider splitting into unit vs E2E |
N_unit == 0 AND total > 0 | High | No unit tests - every change goes through slow layers |
| Only one layer present | Medium | Narrow coverage - regressions in untested layers will slip through |
| Within 50-85% unit / 10-35% integration / 2-20% E2E | OK | Report as balanced |
Report:
## Pyramid
- Unit: <N> (<P>%)
- Integration: <N> (<P>%)
- E2E: <N> (<P>%)
- Verdict: [balanced | inverted | middle-heavy | narrow | unit-absent]
Step 6 - Static anti-pattern checks (T1-T8)
Run per stack. Read ${CLAUDE_SKILL_DIR}/PATTERNS.md for stack-specific grep patterns for all checks below. Checks without a matching pattern produce N/A - skipped for <stack>, not false positives.
T1 - .only / fit / fdescribe committed
Critical. Focused tests in committed code break CI: suite passes locally but skips everything except the focused case. Grep for focused-test markers using patterns from PATTERNS.md → T1.
Report each match with FILE:LINE. If ≥ 2 files have matches, escalate to T8.
T2 - Skipped tests
Medium per match; High if skipped tests > 10% of suite. Grep for skip markers using patterns from PATTERNS.md → T2.
Report: count and top 10 with file:line.
T3 - .todo placeholders
Low. Placeholder tests that never fail. Flag when count grows; not actionable per instance. See PATTERNS.md → T3.
Report: count only.
T4 - Empty test bodies
High. Empty tests always pass and create false confidence. Grep for empty test body patterns from PATTERNS.md → T4 (multiline regex - use multiline: true where supported).
Report: each match with file:line.
T5 - Tests without assertions
High. Test that runs code paths without verifying outcomes.
Heuristic: within a detected test body, grep for at least one of the stack's assertion patterns from PATTERNS.md → T5. Flag any test body that lacks all of its framework's patterns.
Report: each match with file:line. Expect some false positives (helper-only tests, integration setup) - mark findings as probable when the body is very short or wraps another function.
T6 - Hardcoded sleeps in tests
Medium. Sleep-based waits are the primary source of flaky tests. Grep for sleep/delay patterns (numeric sleep ≥ 500ms) using PATTERNS.md → T6.
Report: each match with file:line and duration.
T7 - Debug output left in tests
Low. Doesn't affect correctness but adds noise. Grep test files for debug output patterns from PATTERNS.md → T7.
Report: count only. Do not list individually unless count > 20.
T8 - Multiple .only across files (broken CI)
Critical. Derived from T1: if ≥ 2 distinct files contain T1 matches, the CI gate is broken in a non-obvious way (each file contributes a focused case, so several cases run but most are skipped).
Report: one aggregate finding with the list of affected files.
Step 7 - Report and backlog decision gate
Generate the report using the template in ${CLAUDE_SKILL_DIR}/REPORT.md. Apply the severity guide and backlog writing rules from the same file.
Execution notes
- Do NOT modify any test file. Audit only.
- Do NOT execute tests or the test runner. Coverage reports must already exist on disk.
target:coverage:<path> bypasses auto-detection; accept absolute or project-relative path.- If git history is available, cross-reference C Critical findings against
git diff HEAD~1 (files changed in the most recent commit) to distinguish "0% on pre-existing file" (High) from "0% on file changed in this block" (Critical). If git is not available, downgrade all 0% findings to High.
- This skill complements the Phase 3 test execution (which runs
[TEST_COMMAND] but never inspects the suite itself). Run in Phase 5d Track C after Phase 3 passes.
- After the report, ask: "Do you want me to prepare the corrections for the identified findings?" Reply with
yes only after the user has signed off on the specific findings.
