| name | test-quality |
| kind | evaluator |
| description | Evaluates test code for coverage, assertion quality, isolation, and readability. |
| metadata | {"categories":["test","testing","spec","unit-test","integration-test"],"dimensions":[{"name":"coverage","weight":0.3,"description":"Are the important behaviors and edge cases tested?"},{"name":"assertions","weight":0.3,"description":"Are assertions specific, meaningful, and testing the right thing?"},{"name":"isolation","weight":0.2,"description":"Are tests independent, deterministic, and properly mocked?"},{"name":"readability","weight":0.2,"description":"Are tests clear, well-named, and maintainable?"}]} |
Test Quality Evaluator
Evaluate test code against these criteria.
Coverage (30%)
- Are the happy path scenarios tested?
- Are error paths and failure modes tested?
- Are edge cases covered (empty input, None/null, boundary values, unicode)?
- Are important state transitions tested?
- Is the coverage proportional to risk (critical paths have more tests)?
- Are both positive and negative cases present (testing what should AND should not happen)?
- For bug fixes: is there a regression test that would catch the original bug?
Assertions (30%)
- Are assertions specific (checking exact values, not just "no error")?
- Do assertions test behavior, not implementation details?
- Are error messages in assertions descriptive (what was expected vs what happened)?
- Is each test asserting one logical concept (not testing 5 unrelated things)?
- Are snapshot/golden tests used appropriately (not as a lazy substitute for specific assertions)?
- Do assertions verify side effects where relevant (database state, file writes, API calls)?
- Are floating point comparisons using approximate equality?
Isolation (20%)
- Does each test run independently (no shared mutable state between tests)?
- Are external dependencies mocked or stubbed (network, filesystem, time, randomness)?
- Can tests run in any order without affecting each other?
- Are test fixtures and setup/teardown properly scoped?
- Is time-dependent logic tested with controlled clocks (not real time)?
- Are flaky patterns avoided (sleep-based waits, race conditions)?
- For integration tests: is cleanup reliable (no leaked resources)?
Readability (20%)
- Do test names describe the scenario and expected outcome?
- Is the Arrange/Act/Assert (or Given/When/Then) pattern clear?
- Are test helpers and builders used to reduce boilerplate?
- Is setup code extracted when repeated across multiple tests?
- Are magic numbers and strings explained or named?
- Can a new developer understand what is being tested by reading the test name and body?
- Are tests grouped logically (by feature, by method, by scenario)?
Severity Guide
- Blocker: Tests that always pass (meaningless assertions), tests that depend on external state, tests that mask bugs
- Important: Missing edge case coverage, implementation-coupled tests that break on refactor, flaky tests
- Suggestion: Naming improvements, better test organization, additional helper extraction
