Run any Skill in Manus with one click

audit-benchmark

Stars0

Forks0

UpdatedJune 14, 2026 at 07:13

Produce or refresh one benchmark's AUDIT.md — the standing record of what has actually been checked about it (validity, safety, size, speed, cost, distribution). Use it when adding a benchmark, after changing its Dockerfile or grader, when its AUDIT.md is stale, or as the per-benchmark step of a fleet audit. It fills the machine-measurable fields from real tool runs and leaves human-judgment fields (running, traces-reviewed, replicate-official, cost) explicitly unchecked until a human does them. For the whole-repo sweep use audit-fleet; this skill drives the oracle (core/oracle/README.md) for the grading-soundness rung.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

Exgentic

Exgentic/eval-containers

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software Quality Assurance Analysts and TestersComputer and Mathematical Occupations·SOC 15-1253

File Explorer

2 files

SKILL.md

readonly

name

audit-benchmark

description

Audit a benchmark — write its AUDIT.md

An AUDIT.md is only worth trusting if a ✓ means a check actually ran. This procedure fills the auto-measurable fields from real runs and marks everything else honestly unchecked, so the report never drifts into false confidence. Serves .agents/verification/audit/RULES.md:1–8.

Steps

Start from the template. If benchmarks/<name>/AUDIT.md is absent, copy references/template.md to it. Starting from the template guarantees every dimension is present and every status starts ? (RULES.md:3, :7).
Stamp provenance. Set the frontmatter benchmark, host (e.g. local podman+Rosetta), and commit (git rev-parse --short HEAD). The commit anchors what code was audited; its date and any staleness derive from git, so no separate date is stored (RULES.md:2).
Build it on Rosetta. Build the image per docs/guides/podman-on-apple-silicon.md §6 — single-FROM via DOCKER_BUILDKIT=0 docker build, multi-stage via native podman build --platform linux/amd64 --pull=never. Set building ✓/✗ with the build command as evidence (RULES.md:5, :6, :8).
Oracle it. Run eval-containers oracle <name> [--task-id <id>]. Set oracle ✓ iff gold=1.0 and no-op<1.0, recording both numbers as evidence; n/a for an LLM-judge or stub grader. This is the grading-soundness rung — it proves the grader is neither always-pass nor always-fail (core/oracle).
Read isolation + safety off the artifact. Confirm from the Dockerfile and compose that the benchmark does not undermine the framework's isolation: /tasks stays root-only and the task id never reaches the agent → isolation ✓; the agent runs non-root → agent-nonroot ✓; secrets enter only via --mount=type=secret → secrets-isolated ✓; no agent egress is opened → egress-blocked ✓. Each is inspectable without a run.
Measure size + speed. From the built image and the build/grade runs: image size (and the per-task multiplier for per-task benchmarks) → Size; build, grade, and end-to-end wall-clock → Speed. Copy the measured numbers in; never hand-type an estimate (RULES.md:8).
Check distribution. docker manifest inspect ghcr.io/exgentic/benchmarks/<name>:latest — set published ✓ if it resolves, ✗ if not. Record pull size as the sum of the manifest's layer sizes: the compressed download, distinct from the local image size (RULES.md:6).
Leave the rest honestly unchecked. running (a live agent run), traces-reviewed (human reads a trajectory), replicate-official (a known model reproduces the published score), resource-limited, and cost need work this skill does not do. Mark each ? (or ✗), never ✓, until that work actually happens (RULES.md:5, :7).
Commit it, then refresh the rollup. Commit benchmarks/<name>/AUDIT.md, then regenerate the root project table (the audit-rollup skill) so it reflects this change. The report is a tracked artifact, refreshed whenever the benchmark or its grader changes so its provenance never goes stale (RULES.md:1, :2, :10).

More from this repository

same repository

audit-rules-drift

Exgentic/eval-containers

Walk every normative RULES principle against the actual repository state and report, per rule, whether the code still honours it — compliant, partial drift, or drifted — with file:line evidence and a proposed mechanical check for each gap. Use this for "audit RULES drift", "do the rules still match the code", before a release (step 29 of the `verify` walk), or after a RULES rename that the code may not have followed. This is the rule-text-vs-code consistency pass; for the whole-repo build/fixture/count audit use audit-fleet, and to apply a single topic's rules to one artifact use the meta `review` skill.

2026-06-210

build

Exgentic/eval-containers

Build the Eval Containers image fleet — or any single artifact or eval combination — with `docker buildx bake`, and write the per-artifact `docker-bake.hcl` files the build graph requires. Use when building an image locally, composing a benchmark × agent × model eval, or adding a bake file for a new artifact. For cutting and pushing a tagged release of the whole fleet (with the readiness gate), use the `release` skill, which wraps this one.

2026-06-210

add-benchmark

Exgentic/eval-containers

Use when adding a new benchmark image to the fleet — a Dockerized task plus verifier that evaluates an agent. Walks the directory layout, the five authored files, the two version knobs, required labels, and the three deployment surfaces. For packaging the AI system that runs *inside* a benchmark (install + entrypoint, LLM access), use add-agent instead.

2026-06-170

release

Exgentic/eval-containers

Cut a release of the Eval Containers image fleet — pass the readiness gate, then build, tag, and push every benchmark, agent, model, gateway, and core image to the registry in bulk. Use when tagging a release or pushing the fleet to ghcr.io/exgentic. For building images during local dev (one artifact at a time, no push, no readiness gate), use the `build` skill instead; this skill is the full release flow that wraps it.

2026-06-160

audit-fleet

Exgentic/eval-containers

Run the whole-repository health audit that a release manager reads before cutting a tag — run every mechanical gate, reconcile counts against the README, walk the ten fleet questions (does every benchmark build, does every released benchmark have a fixture, do the docs match disk, are agent versions pinned, is CI real), and classify a red / yellow / green verdict. Use this for "audit the fleet", "is the repo release-ready", a quarterly health check, or as step 25 of the `verify` release walk. This is the whole-repo cross-cutting pass; for a single Dockerfile use audit-dockerfile, for a single trajectory use audit-trajectory, and for RULES-text-vs-code drift use audit-rules-drift.

2026-06-160

audit-trajectory

Exgentic/eval-containers

Run a judgment-level health review of a recorded evaluation trajectory in two halves — was the task the agent saw well-formed (the prompt), and was the actual run sane (refusals, retries, wrong answers, API errors, cost). This is the procedural layer that catches content-filter refusals, factually wrong answers on verifiable benchmarks, and tasks delegated to an external file that the mechanical trajectory rule catalog cannot see. Use this for "audit the fixtures/trajectories", when the mechanical catalog flags a yellow, when a new benchmark batch lands, or as step 24 of the `verify` release walk. This is the per-fixture pass; for per-Dockerfile static health use audit-dockerfile, and for the whole-repo sweep use audit-fleet.

2026-06-160

name

audit-benchmark

description

Audit a benchmark — write its AUDIT.md

Steps

Start from the template. If benchmarks/<name>/AUDIT.md is absent, copy references/template.md to it. Starting from the template guarantees every dimension is present and every status starts ? (RULES.md:3, :7).
Stamp provenance. Set the frontmatter benchmark, host (e.g. local podman+Rosetta), and commit (git rev-parse --short HEAD). The commit anchors what code was audited; its date and any staleness derive from git, so no separate date is stored (RULES.md:2).
Build it on Rosetta. Build the image per docs/guides/podman-on-apple-silicon.md §6 — single-FROM via DOCKER_BUILDKIT=0 docker build, multi-stage via native podman build --platform linux/amd64 --pull=never. Set building ✓/✗ with the build command as evidence (RULES.md:5, :6, :8).
Oracle it. Run eval-containers oracle <name> [--task-id <id>]. Set oracle ✓ iff gold=1.0 and no-op<1.0, recording both numbers as evidence; n/a for an LLM-judge or stub grader. This is the grading-soundness rung — it proves the grader is neither always-pass nor always-fail (core/oracle).
Read isolation + safety off the artifact. Confirm from the Dockerfile and compose that the benchmark does not undermine the framework's isolation: /tasks stays root-only and the task id never reaches the agent → isolation ✓; the agent runs non-root → agent-nonroot ✓; secrets enter only via --mount=type=secret → secrets-isolated ✓; no agent egress is opened → egress-blocked ✓. Each is inspectable without a run.
Measure size + speed. From the built image and the build/grade runs: image size (and the per-task multiplier for per-task benchmarks) → Size; build, grade, and end-to-end wall-clock → Speed. Copy the measured numbers in; never hand-type an estimate (RULES.md:8).
Check distribution. docker manifest inspect ghcr.io/exgentic/benchmarks/<name>:latest — set published ✓ if it resolves, ✗ if not. Record pull size as the sum of the manifest's layer sizes: the compressed download, distinct from the local image size (RULES.md:6).
Leave the rest honestly unchecked. running (a live agent run), traces-reviewed (human reads a trajectory), replicate-official (a known model reproduces the published score), resource-limited, and cost need work this skill does not do. Mark each ? (or ✗), never ✓, until that work actually happens (RULES.md:5, :7).
Commit it, then refresh the rollup. Commit benchmarks/<name>/AUDIT.md, then regenerate the root project table (the audit-rollup skill) so it reflects this change. The report is a tracked artifact, refreshed whenever the benchmark or its grader changes so its provenance never goes stale (RULES.md:1, :2, :10).