regenerate-audit-report

name	regenerate-audit-report
description	Regenerate an OpenAPI public-audit report — re-lint a spec collection with a new thymian version (or a newer spec-repo commit) and produce a fresh dated markdown report, then open a PR. Use when asked to "regenerate the report", "re-run the audit", "update the audit for thymian <version>", "bump thymian and re-lint", or audit a different spec repository with the existing tooling.
tools	Bash, Read, Edit, Write, Grep, Glob

Regenerate the OpenAPI public-audit report

This repo lints large public OpenAPI/Swagger collections with the thymian CLI and commits one dated markdown report per source under reports/<slug>/<YYYY-MM-DD>.md. Spec data and per-spec results are gitignored; only scripts and reports are committed.

Use this skill to produce a new report — almost always because a new thymian version was released, sometimes because you want to re-lint against a fresher commit of the spec repo or point the tooling at a different repo.

The tooling (do not reinvent it)

Path	Role
scripts/fetch-specs.sh	One-time sparse clone of a spec source repo into specs/<slug>/
lint-all-specs.sh	Parallel, resumable linter → one results/<slug>/*.tsv per spec
scripts/aggregate-report.mjs	Aggregates results → reports/<slug>/<date>.md (run automatically at end of a full lint)

README.md documents flags. This skill adds the operational workflow the README does not: version bumps, backing up prior results, verification, and the PR.

The slug is derived from the repo URL: owner/repo → owner-repo, lowercased, non-alphanumerics collapsed to -. Default source is APIs-guru/openapi-directory → slug apis-guru-openapi-directory.

Workflow

1. Confirm the thymian version

thymian version   # e.g. thymian/0.1.11 darwin-arm64 node-v26.0.0

The report records whatever thymian resolves to on PATH. If the user asked for a specific version, verify it is the one installed before linting — re-linting ~4k specs is the expensive step; don't discover a version mismatch afterward. If it's not installed, ask the user to install it (e.g. npm i -g thymian@<version>) rather than guessing.

2. Fetch specs (skipped if already present)

scripts/fetch-specs.sh

Idempotent: it no-ops if specs/<slug>/ exists. To re-lint against a newer commit of the spec repo, delete specs/<slug>/ and specs/<slug>.source.txt first, then re-fetch — the report reads the commit SHA from specs/<slug>.source.txt, so a stale checkout yields a report that misattributes its source.

For a different repo, set the env vars before both scripts (see README "Using a different spec repository"): SPEC_REPO_URL, SPEC_REPO_REF, SPEC_SUBDIR, optionally SPEC_FILE_NAMES.

3. ⚠️ Clear prior results when the version (or spec commit) changed

This is the step that is easy to get wrong. lint-all-specs.sh is resumable: it skips any spec that already has a results/<slug>/<spec>.tsv file. That is exactly what you want to resume an interrupted run of the same version — but if results from a previous thymian version are still there, they will be kept and you'll publish a mixed-version report.

So when the thymian version or the spec-repo commit changed, back up and clear the old results first:

SLUG=apis-guru-openapi-directory   # adjust for other sources
if [ -f results/$SLUG/_meta.txt ]; then
  OLDVER=$(sed -n 's#^thymian_version=thymian/\([^ ]*\).*#\1#p' results/$SLUG/_meta.txt)
  mv results/$SLUG "results/$SLUG.thymian-${OLDVER:-prev}.bak"   # keep old run for comparison
fi

(The .bak directory stays under the gitignored results/, so it is never committed. Keep it — it's handy for diffing version-over-version.)

Resuming a genuinely interrupted run of the same version is the one case where you do not clear: just re-run step 5 and it fills in the gaps. --force re-lints everything but leaves orphaned .tsv files for specs that no longer exist, so prefer clear-and-run over --force for a version bump.

4. Smoke-test the parser first (⚠️ do this on every version bump)

thymian's human-readable lint output is not a stable contract — it changes between versions, and lint-all-specs.sh parses it with awk. When the format changes, the parser silently extracts nothing: specs still get the right status (that comes from the exit code) but errors=0 warnings=0 hints=0 and no per-rule breakdown, so the report looks plausible while being almost entirely empty. (Concretely: 0.1.10 printed Found N errors … with word-based severities like hint:; 0.1.11 prints Summary: N error(s), M warning(s), … with symbol-based severities ✖ ⚠ ℹ. The parser had to be rewritten for that.)

So before spending ~20 minutes on the full run, lint one spec you know has findings and confirm the counts are non-zero and sane:

./lint-all-specs.sh --only 'APIs/adafruit.com/*' --force
grep -h '^META' results/$SLUG/APIs__adafruit*.tsv   # expect errors/warnings/hints > 0, parse_mismatch=0

If the counts are zero (or parse_mismatch=1 shows up across many specs in the final report), the awk block in lint-all-specs.sh — the worker section that reads thymian's stdout into META/RULE TSV lines — no longer matches the output. Run one spec manually (thymian lint --spec openapi:<file> --no-guidance --suppress-feedback --validate-specs), inspect the new format, and update the parser to match before proceeding. Then clear results (step 3) and continue.

5. Run the full lint

./lint-all-specs.sh --jobs <N>     # N ≈ CPU count; leave 1-2 cores free to keep the box responsive

Long-running (thousands of specs). Run it in the background and poll progress rather than blocking:

find results/$SLUG -name '*.tsv' | wc -l    # done so far, vs the "N specs total" the runner printed

Each spec is an independent process; individual failures never abort the run (specs can end up invalid, timeout, crashed — that's data, not a failure). When every discovered spec has a result, the runner auto-runs the aggregator and writes reports/<slug>/<today>.md. If you split the work into --chunks, no single invocation sees all specs finish, so finish with ./lint-all-specs.sh --report-only.

6. Verify the report

Open reports/<slug>/<today>.md and confirm:

• The header cites the expected thymian version.
• The Sources table shows the intended commit/ref and the full spec count.
• The Severity totals and Rules by violation frequency are populated with sane, many-specs-affected numbers — not near-zero. Empty/zero findings on a corpus that clearly has violations means the parser smoke test (step 4) was skipped and the run is bad.
• Data quality does not say "Incomplete run: only X of Y … have results". If it does, results are missing — re-run step 5 (it resumes) until the counts match, then --report-only.
• Sanity-check totals against the previous report (git log/the .bak run) — a wildly different invalid-count or rule list usually means a version behavior change worth calling out in the PR, not a bug.

7. Update the README "Latest report" link

README.md has a **Latest report:** line — point it at the new dated file.

8. Commit and open a PR

Only the report and any doc/tooling changes are committed (specs/ and results/ are gitignored — confirm with git status that no spec data is staged). Branch, commit, push, and open a PR summarizing: thymian version, spec source + commit, and the headline numbers (pass/violation/invalid counts, notable shifts vs. the prior report).

git checkout -b report/<slug>-<today>
git add reports/ README.md            # + any tooling/skill changes
git commit -m "Add <date> audit report for <slug> (thymian <version>)"
git push -u origin HEAD
gh pr create --fill

Troubleshooting

• Report has correct status counts but near-zero Errors/Warnings/Hints and a tiny rule table → the awk parser in lint-all-specs.sh no longer matches this thymian version's output format (see step 4). Fix the parser, clear results, re-lint. This is the most likely failure on a version bump and the easiest to miss.
• Report says "Incomplete run" → not all specs have .tsv results. Re-run step 5 (resumes) or --report-only after all --chunks finish.
• Report shows the old thymian version / mixed numbers → step 3 was skipped; the old results/<slug>/ was reused. Back it up, clear it, re-lint.
• No specs found … → run scripts/fetch-specs.sh first (step 2).
• illegal option -- A / garbled symbols while debugging → macOS cat -A is unsupported; use cat -v, and set LC_ALL=C when byte-matching thymian's UTF-8 severity symbols.
• Regenerate only, no re-lint (e.g. you tweaked the aggregator) → ./lint-all-specs.sh --report-only.
• A new report file per day: the aggregator dates the file by today's date, so two runs on the same day overwrite the same reports/<slug>/<today>.md — intended.