report-regression-check

name	report-regression-check
description	Verify that regenerating cloudsplaining's example report does not silently drop findings. This skill should be used after changing PRIVILEGE_ESCALATION_METHODS or other constants, after editing examples/files/example.json, or before refreshing the committed example fixtures. It snapshots the current report, regenerates, runs a deterministic finding diff, and performs a dogfood visual comparison, failing if any finding disappeared.
allowed-tools	Bash(just:*), Bash(uv run:*), Bash(mkdir:*), Bash(cp:*), Bash(agent-browser:*), Read

Report Regression Check

Regenerating cloudsplaining's example fixtures (utils/example-iam-data.json and cloudsplaining/output/src/sampleData.js) is easy to do but risky: a stale or wrongly configured run can silently DROP findings. This skill guards against that. Additions are expected (new techniques get onboarded); removals are regressions and must block.

When to use

• After changing cloudsplaining/shared/constants.py (e.g. PRIVILEGE_ESCALATION_METHODS).
• After editing examples/files/example.json.
• Before committing any refresh of the committed example fixtures.

Workflow

1. Snapshot the current (old) report

mkdir -p .report-snapshots/old
cp utils/example-iam-data.json .report-snapshots/old/example-iam-data.json
cp index.html .report-snapshots/old/index.html 2>/dev/null || true

.report-snapshots/ is gitignored — never commit it.

2. Regenerate

just generate-report

This rewrites utils/example-iam-data.json and cloudsplaining/output/src/sampleData.js from examples/files/example.json.

3. Deterministic diff (fails on dropped findings)

just compare-reports .report-snapshots/old/example-iam-data.json utils/example-iam-data.json

• Exit 0 → no findings dropped (any additions are listed and are fine).
• Exit 1 → a finding disappeared. STOP and investigate before committing: either the regeneration is wrong, or the change legitimately removes a detection (rare — confirm it is intentional and note it).

4. Dogfood visual comparison

Catch anything the deterministic diff cannot (layout, per-category counts, missing principals/policies). Use the dogfood skill against the freshly generated report (the regenerated root index.html via a file:// URL), with scope:

"Confirm every risk-category section renders, per-category finding counts are at least as high as the old report in .report-snapshots/old/, and no principals or policies present in the old report are missing from the new one."

5. Summarize

Report findings added, findings removed (should be zero), and the dogfood result. If step 3 exited non-zero or dogfood found missing data, the regeneration is NOT safe to commit.

Notes

• The deterministic diff keys findings by (json-path, finding-id) and is reliable for incremental regenerations (policy IDs in example.json are stable). If example.json itself is restructured so IDs change, everything appears added/removed — review manually.
• This skill never commits the .report-snapshots/ snapshots.
• See utils/compare_example_reports.py for the diff implementation.