| name | audit-rollup |
| description | Generate or refresh containers/AUDIT.md — the project-level table that shows the bottom line of every benchmark's audit in one grid. Use it after any benchmark's AUDIT.md changes, or to see fleet-wide audit status at a glance. It scrapes each benchmarks/<name>/AUDIT.md and copies status verbatim, never inventing status the source files don't carry. For one benchmark's report use audit-benchmark. |
Generate the project audit table
The containers/AUDIT.md rollup is the one place to see, across the fleet, how far each
benchmark has actually been audited. It is generated, never hand-edited, so it
cannot drift from the per-benchmark reports it summarizes. Serves
.agents/verification/audit/RULES.md:9 and :10.
Steps
-
Collect. Read every benchmarks/<name>/AUDIT.md — its commit and each
validity and safety status. A benchmark with no AUDIT.md is a row of ?; its
absence is itself a bottom line worth showing (RULES.md:9).
-
Roll up safety. Reduce the four safety checks to one column: ✓ only if
all four are ✓, ✗ if any is ✗, else ?. The table shows the headline;
the per-benchmark file holds the detail.
-
Resolve Published live. Ask the registry which benchmark images exist —
one gh api /orgs/Exgentic/packages?package_type=container call (filter
benchmarks/<name>), or docker manifest inspect ghcr.io/exgentic/benchmarks/<name>.
Set Published ✓ if present, ✗ if not. Compute it live, like the Audited
date — so the column is complete even for benchmarks with no AUDIT.md (RULES.md:9).
-
Render the table. From references/project-template.md, write one row per
benchmark — Building, Running, Isolation, Oracle, Traces, Replicate, Safety,
Published, Audited — copying each status verbatim from its source. The Audited
cell is the commit's date (git show -s --format=%cs <commit>), not a stored
field. Never upgrade a ? to a ✓ (RULES.md:10); the rollup is only as audited
as its sources.
-
Flag stale rows. For each benchmark, compare its AUDIT.md commit to the
latest commit touching its sources — git log -1 --format=%H -- benchmarks/<name>/ ':(exclude)benchmarks/<name>/AUDIT.md'. Exclude the report
itself, or committing an AUDIT.md would mark its own benchmark stale. If the
sources changed since, mark its Audited cell stale (⚠): a green status against
old code is worse than none (RULES.md:11).
-
Total each column. Append a line counting ✓ per column (e.g. oracle
X/N, published Y/N), so the fleet's audit and publish progress read off the bottom.
-
Write to the root. Write the table to ./AUDIT.md and commit it.
Regenerate it whenever any benchmark's AUDIT.md changes, so the rollup and
its sources never disagree (RULES.md:10).
