writing-ratchet-tests

name	writing-ratchet-tests
description	Write ratchet tests to prevent accumulation of code anti-patterns. Use when asked to create a "ratchet test" for tracking and preventing specific code patterns (e.g., TODO comments, inline imports, broad exception handling).

Writing Ratchet Tests

This skill provides guidelines for writing ratchet tests that prevent accumulation of code anti-patterns in a project.

What are Ratchet Tests?

Ratchet tests are a testing pattern that:

• Track the current count of a specific anti-pattern in the codebase
• Prevent that count from increasing (using inline-snapshot)
• Allow the count to decrease (improvement is always allowed)
• Provide clear, actionable feedback when violations increase

Common use cases:

• TODO comments
• Inline imports
• Use of eval() or exec()
• Broad exception handling (bare except, except Exception)
• Any other code pattern you want to gradually eliminate

Architecture

The ratchet system has three layers:

1. Rule definitions in libs/imbue_common/imbue/imbue_common/ratchet_testing/common_ratchets.py -- RegexRatchetRule and RatchetRuleInfo objects
2. Wrapper functions in libs/imbue_common/imbue/imbue_common/ratchet_testing/standard_ratchet_checks.py -- one function per ratchet rule
3. Test functions in each project's test_ratchets.py -- call the wrapper with a snapshot count

All projects must have the same set of test functions (enforced by test_meta_ratchets.py).

Adding a New Common Ratchet

Step 1: Define the Rule

Add a RegexRatchetRule to common_ratchets.py:

PREVENT_MY_PATTERN = RegexRatchetRule(
    rule_name="my pattern usages",
    rule_description="Explain why this pattern is problematic and what to do instead",
    pattern_string=r"my_regex_pattern",
    is_multiline=False,  # set True if using ^ or $ anchors
)

Or a RatchetRuleInfo for AST-based checks (add the detection function to ratchets.py).

Step 2: Add a Wrapper Function

Add to standard_ratchet_checks.py (the single source of truth for which ratchets exist):

def check_my_pattern(source_dir: Path, max_count: int) -> None:
    assert_ratchet(PREVENT_MY_PATTERN, source_dir, max_count)

Remember to import the rule at the top of the file. The function name determines the test name: check_foo becomes test_prevent_foo.

Step 3: Sync and Set Counts

uv run python scripts/sync_common_ratchets.py
uv run pytest --inline-snapshot=update -k test_ratchets

The sync script reads standard_ratchet_checks.py, generates test_prevent_my_pattern in all 26 test_ratchets.py files with snapshot(0), then the pytest command sets the actual violation counts per project.

Adding a Project-Specific Ratchet

Not all ratchets belong in every project. If a ratchet only applies to one project (e.g., a project-specific API convention), add it to a separate file -- NOT to test_ratchets.py (which must define the same test function names across all projects, enforced by test_meta_ratchets.py).

Create a project-specific ratchet test file (e.g., test_project_ratchets.py) using the core API directly:

from pathlib import Path

from inline_snapshot import snapshot

from imbue.imbue_common.ratchet_testing.core import FileExtension
from imbue.imbue_common.ratchet_testing.core import RegexPattern
from imbue.imbue_common.ratchet_testing.core import check_regex_ratchet
from imbue.imbue_common.ratchet_testing.core import format_ratchet_failure_message

_DIR = Path(__file__).parent.parent.parent


def test_prevent_my_project_pattern() -> None:
    pattern = RegexPattern(r"my_regex_pattern", multiline=False)
    chunks = check_regex_ratchet(_DIR, FileExtension(".py"), pattern)
    assert len(chunks) <= snapshot(), format_ratchet_failure_message(
        rule_name="my project pattern",
        rule_description="Why this is problematic and what to do instead",
        chunks=chunks,
    )

Run with --inline-snapshot=create to set the initial count, then verify it passes normally.

Important Rules

• Never add per-project code quality ratchets to test_meta_ratchets.py. Meta ratchets are for repo-wide structural checks only.
• If a ratchet applies to all projects, use the common ratchet workflow (sync script). If it only applies to one project, use a separate test file with the core API.
• Keep test function names descriptive: test_prevent_<anti_pattern>
• Provide clear rule_description that explains WHY the pattern is bad and WHAT to do instead
• Never blindly update snapshots -- investigate why a count increased

Troubleshooting

Pattern not matching expected violations:

• Check if you need multiline=True for patterns using ^ or $
• Verify the regex is correct using a regex tester
• Check that the file extension is correct
• Ensure violations are in git-tracked files (git blame only works on committed code)

Test fails after running:

• This is expected if the current count is higher than the snapshot
• Always fix the violations if a ratchet test fails--it's because you messed something up. NEVER run with --inline-snapshot=fix
• Never just blindly update snapshots - investigate why the count increased

Snapshot shows 0 but violations exist:

• The regex pattern might be incorrect
• Try running with multiline=True if using ^ or $ anchors