Record executed commands, outputs, and recommendations.
→ See: imbue:proof-of-work
Test Quality Checklist (Condensed)
Clear test structure (Arrange-Act-Assert)
Critical paths covered (auth, validation, errors)
Specific assertions with context
No flaky tests (dead waits, order dependencies)
Reusable fixtures/factories
Invariant-encoding tests intact (see below)
Invariant-Encoding Tests
Tests encode design invariants as well as verifying behavior.
A test that asserts "module A never imports from module B"
encodes a layer boundary. A test that
asserts "this function is pure" encodes a concurrency
model. These tests are load-bearing in ways that
coverage metrics cannot capture.
During review, check:
Were invariant-encoding tests removed or weakened?
A test that enforced an architectural boundary,
data structure constraint, or API contract should
not be deleted without naming the invariant being
abandoned and escalating to human judgment.
Were test expectations changed to match a broken
implementation? If an assertion value changed, ask:
did the requirement change, or did the agent change
the test to make its code pass? The latter is the
single most dangerous form of test tampering.
Are new invariants encoded as tests? When a design
decision is made (choice of data structure, module
boundary, error strategy), there should be at least
one test whose failure would signal that the
invariant was violated.
Red flag patterns:
Pattern
Risk
@pytest.mark.skip added to a passing test
Invariant being silently dropped
Assertion changed from specific to broad
Constraint being relaxed
Test renamed to describe new behavior
Old invariant erased from history
Test deleted "because it tested old code"
Invariant removed without replacement
When invariant erosion is detected:
Do NOT approve. Flag as a BLOCKING quality issue and
present the three options to the human:
Preserve: Revert the test change, fix the
implementation to satisfy the invariant
Layer: Keep the invariant test, add the new
behavior alongside it (accepting inelegance)
Revise: The invariant is genuinely wrong; remove
the old test AND write a new test encoding the
replacement invariant
This is a judgment call that models get wrong far too
often. Default to option 1 (preserve) when no human is
available.
Drop or label UNVERIFIED any finding the verifier fails (exit 1); only
verified findings enter the report. See Skill(imbue:review-core) Step 5
and Skill(imbue:structured-output) for the schema.
Exit Criteria
Frameworks detected and documented
Coverage analyzed and gaps identified
Scenario quality assessed
Remediation plan created with owners and dates
Evidence logged with citations
Every reported finding carries a Location + verbatim Anchor confirmed
by citation_verifier.py (exit 0), or unverified findings were dropped
or labeled UNVERIFIED
Troubleshooting
Common Issues
Tests not discovered
Ensure test files match pattern test_*.py or *_test.py. Run pytest --collect-only to verify.
Import errors
Check that the module being tested is in PYTHONPATH or install with pip install -e .
Async tests failing
Install pytest-asyncio and decorate test functions with @pytest.mark.asyncio