Menu
Scan test files for anti-patterns including mesa-optimization, disabled tests, trivial assertions, and error swallowing
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | Test Quality Audit |
| description | Scan test files for anti-patterns including mesa-optimization, disabled tests, trivial assertions, and error swallowing |
| category | validation |
| usage | qa, action, backend |
| version | 1 |
| created | "2025-11-05T00:00:00.000Z" |
| converted_from | docs/agents/shared-ref-docs/test-quality-red-flags.md |
Use this skill when you need to:
Triggers:
Red Flags to Watch For:
For code review (PR-specific):
For systematic audit (codebase-wide):
*.test.js, *.spec.ts, *_test.py, test_*.pyRun the following checks on test files:
Purpose: Detect tests that are skipped or disabled in committed code
Patterns to detect:
// JavaScript/TypeScript:
describe.skip("...", ...)
it.skip("...", ...)
test.skip("...", ...)
it.only("...", ...) // CRITICAL: Means other tests are ignored
test.only("...", ...)
// Python:
@unittest.skip("...")
@pytest.mark.skip
pytest.skip()
# TODO: fix this test
Scan command:
# JavaScript/TypeScript:
grep -rn -E "\.(skip|only)\(" tests/ spec/ __tests__/
# Python:
grep -rn -E "(@unittest\.skip|@pytest\.mark\.skip|pytest\.skip\(|# TODO.*test)" tests/
Severity:
.only() in committed code (all other tests ignored).skip() without issue reference or justification.skip() with TODO comment but no timelinePass Criteria: No disabled tests, OR all disabled tests have:
# LAW-123: Re-enable when feature X ships)Purpose: Detect assertions that don't validate meaningful behavior
Patterns to detect:
// JavaScript/TypeScript:
expect(true).toBe(true)
expect(result).toBeDefined()
expect(response).toBeTruthy()
expect(error).toBeFalsy() // Error swallowing!
// Python:
assert True
assert result is not None
assert response # Vague assertion
Scan command:
# JavaScript/TypeScript:
grep -rn -E "(expect\(true\)|expect\(false\)|\.toBeTruthy\(\)|\.toBeFalsy\(\)|\.toBeDefined\(\))" tests/ spec/
# Python:
grep -rn -E "(assert True|assert False|assert [a-zA-Z_]+ is not None)" tests/
Severity:
expect(true).toBe(true))toBeDefined() without validating actual valuePass Criteria: All assertions validate specific expected values or behaviors
Purpose: Detect try/catch blocks that suppress errors without assertions
Patterns to detect:
// JavaScript/TypeScript:
try {
// ... test code ...
} catch (error) {
// No assertion on error - swallowed!
}
// Broad catch without validation:
try {
await riskyOperation()
} catch (e) {
console.log(e) // Logged but not asserted
}
# Python:
try:
# ... test code ...
except Exception:
pass # Error swallowed!
# Broad except without assertion:
try:
risky_operation()
except Exception as e:
print(e) # Logged but not asserted
Manual Review Required: This pattern requires reading test code context
Check for:
catch (Exception) is suspicious)Severity:
Pass Criteria: All try/catch blocks either:
Exception catch)Purpose: Detect HTTP calls replaced with mocks/constants without rationale
Patterns to detect:
// JavaScript/TypeScript:
// const response = await fetch('/api/endpoint')
const response = { status: 200, data: mockData }
// await api.createUser(userData)
// Commented out actual API call
Scan command:
# Look for commented HTTP calls:
grep -rn -E "// .*(fetch\(|axios\.|http\.|api\.)" tests/ spec/
Severity:
Pass Criteria: All mocked HTTP calls either:
Purpose: Detect tests weakened to make them pass (instead of fixing code)
Manual Review Required: Compare test changes in PR diff
Patterns to detect:
Check in PR diff:
- expect(result.users).toHaveLength(5)
+ expect(result.users).toHaveLength(3) // Why changed? Bug or feature?
- expect(response.status).toBe(200)
+ // Status check removed - why?
- expect(() => validateInput('')).toThrow('Input required')
+ expect(() => validateInput('')).not.toThrow() // Validation removed?
Severity:
Pass Criteria: All test weakening changes are justified with:
Purpose: Detect security validation bypassed via ignore patterns
Patterns to detect:
// eslint-disable security/detect-non-literal-fs-filename
// eslint-disable-next-line security/detect-unsafe-regex
// prettier-ignore
Scan command:
# Look for security-related linter disables:
grep -rn -E "(eslint-disable.*security|nosec|# noqa.*security)" tests/ src/
Severity:
Pass Criteria: All security linter disables have:
Organize all detected anti-patterns by severity:
CRITICAL (blocks merge):
.only() in committed tests (all other tests ignored)HIGH (requires fix before approval):
MEDIUM (request fix, but can approve with warning):
INFO (feedback for improvement):
If anti-patterns found, generate a report:
**Test Quality Audit Results**
⚠️ **ISSUES FOUND** - Test quality concerns detected
### Critical Issues (Must Fix Before Merge)
1. **Disabled Test with .only()** (tests/user.test.ts:45):
- Pattern: `it.only("creates user", ...)`
- Issue: All other tests in suite are ignored
- Fix: Remove `.only()` or explain why only this test should run
2. **Empty Catch Block** (tests/api.test.ts:89):
- Pattern: `catch (error) { /* empty */ }`
- Issue: Errors swallowed without assertion
- Fix: Assert on expected error or remove try/catch
### High Priority Issues (Fix Recommended)
3. **Disabled Test Without Justification** (tests/auth.test.ts:120):
- Pattern: `it.skip("validates token expiry", ...)`
- Issue: No Linear issue or explanation for skip
- Fix: Add issue reference or re-enable test
4. **Trivial Assertion** (tests/validation.test.ts:67):
- Pattern: `expect(true).toBe(true)`
- Issue: Assertion doesn't validate actual behavior
- Fix: Assert on specific validation result
### Medium Priority Issues (Warnings)
5. **Vague Assertion** (tests/response.test.ts:34):
- Pattern: `expect(response).toBeDefined()`
- Issue: Doesn't validate response contents
- Fix: Assert on specific response fields (status, data, etc.)
**Recommendation**: [BLOCKED | REQUEST FIXES | APPROVED WITH WARNINGS]
If audit passes, confirm quality:
**Test Quality Audit Results**
✅ **PASSED** - No test quality issues found
All checks passed:
- [x] No disabled tests without justification
- [x] All assertions validate specific behaviors
- [x] Error handling includes assertions
- [x] No HTTP calls replaced with inline mocks
- [x] No test weakening detected
- [x] Security linter rules properly applied
**Recommendation**: APPROVED for merge
// Bad example:
it.skip("validates email format", () => {
// Test disabled, no explanation why
})
// Good example:
// LAW-456: Re-enable when email validation RFC compliance added
it.skip("validates email format per RFC 5322", () => {
// Test disabled with clear reference to tracking issue
})
// Bad example:
it("creates user", async () => {
const result = await createUser(userData)
expect(result).toBeDefined() // Vague - what about result?
})
// Good example:
it("creates user", async () => {
const result = await createUser(userData)
expect(result.id).toBeDefined()
expect(result.email).toBe(userData.email)
expect(result.status).toBe("active")
})
// Bad example:
it("handles invalid input", async () => {
try {
await processInput(null)
} catch (error) {
console.log(error) // Logged but not asserted
}
})
// Good example:
it("handles invalid input", async () => {
await expect(processInput(null)).rejects.toThrow("Input cannot be null")
})
// Bad example:
it("fetches user data", async () => {
// const response = await fetch('/api/users/123')
const response = { id: 123, name: "Test User" } // Inline mock
expect(response.name).toBe("Test User")
})
// Good example:
it("fetches user data", async () => {
// Mock at framework level, not inline
jest.spyOn(api, "getUser").mockResolvedValue({ id: 123, name: "Test User" })
const response = await fetchUserData(123)
expect(response.name).toBe("Test User")
expect(api.getUser).toHaveBeenCalledWith(123)
})
grep: Pattern matching for anti-pattern detectiondocs/agents/qa/qa-agent.md (Test Quality Standards section)/security-validate - Security validation patterns/test-standards - Comprehensive test quality validation (if available)test-audit-protocol.md - Comprehensive test audit proceduresScan for disabled tests:
# JavaScript/TypeScript:
grep -rn -E "\.(skip|only)\(" tests/ spec/ __tests__/
# Python:
grep -rn -E "(@unittest\.skip|@pytest\.mark\.skip|pytest\.skip\()" tests/
Scan for trivial assertions:
# JavaScript/TypeScript:
grep -rn -E "(expect\(true\)|expect\(false\)|\.toBeTruthy\(\)|\.toBeFalsy\(\)|\.toBeDefined\(\))" tests/
# Python:
grep -rn -E "(assert True|assert False)" tests/
Scan for commented HTTP calls:
grep -rn -E "// .*(fetch\(|axios\.|http\.|api\.)" tests/
Scan for security linter suppression:
grep -rn -E "(eslint-disable.*security|nosec|# noqa.*security)" tests/ src/
Create complete Claude Code agents for the Traycer enforcement framework. This skill should be used when creating new agents or updating existing agents. Creates all agent components including system prompt, hooks, skills, commands, scripts, and reference docs. Also updates coordination documents that list available agents. Based on Anthropic's official agent development best practices.
Clean and normalize a large backlog of GitHub pull requests in messy repositories. This skill should be used when facing a repository with numerous stale, conflicted, or outdated PRs that need systematic review, conflict resolution, testing, and disposition (merge, close, or defer). Handles PR triage, automated rebasing, CI verification, risk assessment, and safe merging while maintaining default branch stability.
This skill should be used when the user requests to create professional business documents (proposals, business plans, or budgets) from templates. It provides PDF templates and a Python script for generating filled documents from user data.
This skill should be used when writing documentation for codebases, including README files, architecture documentation, code comments, and API documentation. Use this skill when users request help documenting their code, creating getting-started guides, explaining project structure, or making codebases more accessible to new developers. The skill provides templates, best practices, and structured approaches for creating clear, beginner-friendly documentation.
Define custom Claude Code slash commands for agents in the Traycer enforcement framework. This skill should be used when creating or updating agents and needing to specify reusable prompts that agents can execute as slash commands. Commands are Markdown files stored in .claude/commands/ and referenced in agent config.yaml files. This is for Claude Code slash commands (/command-name), not bash/CLI commands.
This skill should be used when analyzing CSV datasets, handling missing values through intelligent imputation, and creating interactive dashboards to visualize data trends. Use this skill for tasks involving data quality assessment, automated missing value detection and filling, statistical analysis, and generating Plotly Dash dashboards for exploratory data analysis.