crucible-audit

name	crucible-audit
description	Forensic test-quality audit that detects test fraud — hollow assertions, coverage-omit gaming, mock proliferation, dead/gated tests, untested entry points, and README badge inflation that produce false green signals. Use when asked to "run a CRUCIBLE audit", "check/audit test quality", "find silent failures", "find hollow tests", "audit assertions/mocks/coverage config", "why do tests pass but nothing works", or before any release gate (pre-publish/pre-deploy) when a high test count or coverage % needs verifying against reality.
argument-hint	[path]
allowed-tools	Read, Grep, Glob, Bash

CRUCIBLE Test Quality Audit

Overview

The CRUCIBLE Audit skill provides the methodology for forensic analysis of test suites. It detects test fraud — the pattern where test count and coverage numbers look healthy but the tests prove nothing about whether the software actually works.

Origin: Wolf's forensic audit of Podcast-Pipeline (2026-03-07). 1,601 tests, 77% claimed coverage — actual coverage ~15%, nothing worked when a human used it.

Language Matrix (read this BEFORE running any detection command)

The detection greps below are Python/pytest-shaped (tests/, def test_, assert x is not None). Run them unchanged against a JS/TS or Rust repo and every pattern returns 0 → a false CLEAN verdict. Always translate to the stack first. NXTG-Forge's own three repos are all non-Python.

Ecosystem	Test dirs / files	Test decl	Hollow-assertion tells	Mock tells	Coverage config
Python / pytest	tests/, test_*.py	def test_	assert x is not None, .exists(), assert True, len >= 0/1, isinstance	@patch, MagicMock, Mock()	[tool.coverage.run] omit, .coveragerc
Node / vitest / jest	tests/, __tests__/, *.test.{mjs,ts,tsx}	it(, test(	toBeDefined, toBeTruthy, not.toBeNull, expect(x).toBeDefined()	vi.mock, jest.mock, vi.fn()	coverage.exclude, coveragePathIgnorePatterns, collectCoverageFrom
Rust / cargo	inline #[test] in src/**, tests/	#[test], #[tokio::test]	assert!(x.is_some()), assert!(x.is_ok()), assert!(true)	mockall, #[automock], hand-rolled stub structs	tarpaulin --exclude-files, #[cfg(not(coverage))]

When $1 (the audit path) points at a repo, detect the stack first (ls for Cargo.toml / package.json / pyproject.toml), then use the matching row. Report the stack in the audit header.

The 8 Fraud Patterns

Pattern 1: Coverage Omit Gaming

What: Excluding hard-to-test code from [tool.coverage.run] omit to inflate coverage %. Detection:

# Python: check pyproject.toml / setup.cfg / .coveragerc
grep -A 20 '\[tool.coverage.run\]' pyproject.toml | grep -i omit
grep -A 20 '\[run\]' .coveragerc | grep -i omit

# JavaScript: check jest.config / vitest.config
grep -A 10 'coveragePathIgnorePatterns\|collectCoverageFrom' jest.config* vitest.config*

Severity: CRITICAL — this is deliberate metric manipulation. Remediation: Remove omit entries. All source code counts. Report real numbers.

Pattern 2: Hollow Assertions

What: Assertions that pass regardless of correctness. Detection:

# Count trivially-passing assertions
grep -rn "assert.*is not None" tests/ | wc -l
grep -rn "assert.*\.exists()" tests/ | wc -l
grep -rn "assert True" tests/ | wc -l
grep -rn "assert len.*>= 0" tests/ | wc -l
grep -rn "assert len.*>= 1" tests/ | wc -l  # May be valid, needs context
grep -rn "assert isinstance" tests/ | wc -l  # Type check without value check

Severity: HIGH — creates false confidence. Remediation: Each assertion must be FALSIFIABLE by a realistic bug. If the assertion can't fail when the code is broken, it's hollow.

Pattern 3: Mock Proliferation

What: Mocking internal components so tests verify mock behavior, not real behavior. Detection:

# Count mock usage
grep -rn "@patch\|@mock" tests/ | wc -l
grep -rn "MagicMock\|Mock()" tests/ | wc -l
grep -rn "patch(" tests/ | wc -l

# Mock ratio: mocks / total test functions
MOCKS=$(grep -rn "patch\|Mock\|MagicMock" tests/ | wc -l)
TESTS=$(grep -rn "def test_" tests/ | wc -l)
echo "Mock ratio: $MOCKS mocks / $TESTS tests = $(echo "scale=1; $MOCKS * 100 / $TESTS" | bc)%"

Severity: HIGH if >30% mock ratio. CRITICAL if integration/e2e tests use mocks. Standard: Mocks are for external APIs you don't control. Internal code uses real implementations. Red flag: Files named *integration* or *e2e* or *smoke* that contain @patch or MagicMock.

Pattern 4: Dead Test Infrastructure

What: Tests gated by environment variables, markers, or conditions that are never activated. Detection:

# Find env-gated tests
grep -rn "skipIf\|skipUnless\|PYTEST_\|os.environ.get" tests/ | grep -i "skip\|pytest"
grep -rn "@pytest.mark.skip" tests/

# Check if the gate vars are ever set
for var in $(grep -roh 'os.environ.get("[^"]*")' tests/ | sort -u | sed 's/os.environ.get("//;s/")//' ); do
    grep -rn "$var" Makefile CI/ .github/ pytest.ini pyproject.toml 2>/dev/null || echo "DEAD: $var never set"
done

Severity: HIGH — dead tests inflate count without providing coverage. Remediation: Either enable the gate in CI, or delete the tests. Tests that can't run aren't tests.

Pattern 5: Mock-Heavy Integration Tests

What: Tests labeled "integration" or "e2e" that mock the components they claim to integrate. Detection:

# Check integration/e2e test files for mocks
for f in $(find tests/ -name "*integration*" -o -name "*e2e*" -o -name "*smoke*"); do
    MOCKS=$(grep -c "patch\|Mock\|MagicMock" "$f" 2>/dev/null || echo 0)
    TESTS=$(grep -c "def test_" "$f" 2>/dev/null || echo 0)
    echo "$f: $MOCKS mocks in $TESTS tests"
done

Severity: CRITICAL — the most dangerous fraud because it appears to be the highest-value testing. Remediation: Real integration tests use real components. If you can't run the real component in CI, the test is a unit test — label it honestly.

Pattern 6: Untested Entry Points

What: User-facing code (API, CLI, main) with near-zero coverage. Detection:

# Python: check coverage of entry point files
python -m pytest --cov=src --cov-report=term-missing 2>&1 | grep -E "(api|cli|main|__main__|app)\." | grep -v "100%"

# JavaScript: check coverage of route/handler files
npx jest --coverage 2>&1 | grep -E "(route|handler|controller|app)\." | grep -v "100"

Severity: HIGH — if users can't reach tested code, tests don't matter.

Pattern 7: No Output Quality Verification

What: Tests that check "did the function run?" but not "is the output correct?" Detection: Read 5-10 test files and check: does any assertion verify OUTPUT QUALITY?

• Audio: RMS levels, duration, frequency analysis
• Text: content accuracy, formatting correctness
• Data: value ranges, relationships, invariants
• API: response body content (not just status codes)

If assertions only check: file exists, count > 0, no exception raised, status 200 — that's Pattern 7. Severity: HIGH — the fundamental purpose of testing is to verify correctness.

Pattern 8: README Badge Fraud

What: README badges showing different numbers than reality. Detection:

# Compare badge claims to actual
grep -oP 'tests-\K[0-9]+' README.md
grep -oP 'coverage-\K[0-9]+' README.md
python -m pytest --tb=no -q 2>&1 | tail -1
python -m pytest --cov=src --cov-report=term 2>&1 | grep TOTAL

Severity: MEDIUM — but erodes trust in all project reporting.

The Audit Report Template

## CRUCIBLE TEST QUALITY AUDIT — {PROJECT_NAME}

### Verdict: {PASS | FAIL | CRITICAL FAIL}

| Pattern | Status | Count | Severity |
|---------|--------|-------|----------|
| 1. Coverage omit gaming | {CLEAN/FOUND} | {N entries} | {severity} |
| 2. Hollow assertions | {CLEAN/FOUND} | {N instances} | {severity} |
| 3. Mock proliferation | {CLEAN/FOUND} | {ratio}% | {severity} |
| 4. Dead test infrastructure | {CLEAN/FOUND} | {N tests} | {severity} |
| 5. Mock-heavy integration | {CLEAN/FOUND} | {N files} | {severity} |
| 6. Untested entry points | {CLEAN/FOUND} | {N LOC} | {severity} |
| 7. No quality verification | {CLEAN/FOUND} | {0 tests} | {severity} |
| 8. README badge fraud | {CLEAN/FOUND} | {delta} | {severity} |

### Real Numbers
- Tests collected: {N}
- Tests actually passing: {N}
- Tests skipped/dead: {N}
- Claimed coverage: {N}%
- Actual coverage (no omits): {N}%
- Mock ratio: {N}%
- Integration tests using mocks: {N}/{total}

### Smoking Gun Examples
{3-5 specific examples with file paths and line numbers}

### Remediation Priority
1. {highest impact fix}
2. {next fix}
3. {next fix}

When to Use This Skill

• After any project claims high test count or coverage
• Before any release gate (pre-publish, pre-deploy)
• When a Human Oracle finds issues that tests should have caught
• During CRUCIBLE Protocol enforcement
• When test count increases rapidly (>100 tests in one sprint = suspicious)
• During portfolio enrichment cycles (spot audits)

Worked Example (Podcast-Pipeline, the origin audit)

# Stack detect → Python. Path = ./ (the whole repo).
grep -A20 '\[tool.coverage.run\]' pyproject.toml | grep -i omit
#   omit = ["src/engines/*", "src/ml/*", "src/api/*"]   ← core product excluded  → Pattern 1, P0
MOCKS=$(grep -rn "@patch\|MagicMock\|Mock()" tests/ | wc -l)   # 439
TESTS=$(grep -rn "def test_" tests/ | wc -l)                   # 1601
# ratio 27% overall, BUT: test_pipeline_e2e.py mocks TTS + ASR + ffmpeg  → Pattern 5, CRITICAL
grep -rn "assert.*is not None\|\.exists()\|assert len.*>= 1$" tests/ | wc -l   # 153  → Pattern 2, HIGH

Reported: 1,601 tests, 77% coverage. After removing unjustified omits and re-running: ~15% real coverage. Verdict: CRITICAL FAIL — fraud margin 62 points. The three things the pipeline exists to do (TTS/ASR/ffmpeg) were all mocked in the one test named e2e.

Gotchas

• The greps are Python-shaped — a JS/TS/Rust repo returns 0 for every pattern and looks CLEAN. This is the #1 false-negative. Always use the Language Matrix row for the actual stack. All three NXTG-Forge repos (forge-ui = vitest/TS, governance-mcp = Node, forge-orchestrator = Rust) would pass the Python greps with a fraudulent clean bill.
• Nonstandard test dir → silent 0. Detection assumes tests/. Repos use test/, spec/, __tests__/, or inline (src/**/*.test.ts, Rust #[test] in src/). find/grep hitting a missing tests/ returns nothing, not an error. ls the real layout first.
• Grep counts are LEADS, not verdicts. grep | wc -l counts matching lines (multiple mocks on one line undercount; a comment or string containing "Mock"/"assert True" overcounts). assert len >= 1 and assert isinstance are sometimes legitimate. Every flagged count must be confirmed by reading the lines (Phase 2) before it enters the report — never accuse on a raw count.
• bc may be absent (minimal containers, some CI images) — the mock-ratio one-liner then prints nothing. Fall back to integer math or awk.
• Check 8.5 mutates config — this is the one write in a read-only audit. To get the real coverage number you temporarily strip omit/exclude and re-run. RESTORE the config afterward (or edit a copy). The "read-only" rule is about not touching source/tests, not about never running the suite.
• Running the suite has side effects. pytest --cov / vitest --coverage executes code — it can hit a DB, write coverage artifacts, or run migrations. On an unknown repo, prefer static greps first; only run the suite when you understand what it touches.
• A green suite is not evidence of correctness — that is the whole premise. dx3 (FP-006) had 3,277 passing tests while graph queries silently returned empty. Never let "all tests pass" close a Pattern-7 finding.

The Fundamental Test

If you deleted all mocks, removed all env-var gates, and ran the full test suite against real infrastructure — how many tests would pass?

That number is the REAL test count. Everything above it is scaffolding. Scaffolding is fine, but it must never be confused with the real thing.

Additional resources

• Full fraud-pattern catalog (FP-001…FP-008, with TS/Node signatures, git-log detection, and the dx3 silent-exception + test-count-inflation + mock-tautology patterns not covered above): references/forensic-patterns.md. Extend it — assign the next FP-NNN — whenever a new pattern surfaces in a portfolio audit.