regression-consistency-checker

name

regression-consistency-checker

description

Checks whether a new version of a repository preserves the behavior observed by tests on the old version. Use this skill when comparing two versions of code to detect regressions, verify refactoring safety, validate bug fixes don't break existing functionality, or ensure backward compatibility. Detects differences in function outputs, exceptions, observable states, and performance between versions. Generates reports highlighting potential regressions (critical, high, medium, low severity), improvements, and areas requiring verification. Triggers when users ask to check for regressions between versions, compare test behavior across versions, verify behavior preservation, or validate that changes don't break existing tests.

Regression Consistency Checker

Check whether a new version of a repository preserves the behavior observed by tests on the old version.

Workflow

1. Prepare Versions

Set up old version:

# Tag or note the old version
git tag old-version

# Or checkout specific commit
git checkout <old-commit-hash>

Set up new version:

# Tag the new version
git tag new-version

# Or checkout new commit
git checkout <new-commit-hash>

Ensure clean environment:

• Same dependencies installed
• Same test configuration
• Same environment variables
• Deterministic test execution (fix random seeds, mock time)

2. Run Tests on Old Version

Capture baseline results:

# Python (pytest with JSON report)
git checkout old-version
pytest --json-report --json-report-file=old_results.json


git checkout old-version
npm  -- --json --outputFile=old_results.json


pytest --json-report --json-report-file=old_results_1.json
pytest --json-report --json-report-file=old_results_2.json