eval-containers

Walk every normative RULES principle against the actual repository state and report, per rule, whether the code still honours it — compliant, partial drift, or drifted — with file:line evidence and a proposed mechanical check for each gap. Use this for "audit RULES drift", "do the rules still match the code", before a release (step 29 of the `verify` walk), or after a RULES rename that the code may not have followed. This is the rule-text-vs-code consistency pass; for the whole-repo build/fixture/count audit use audit-fleet, and to apply a single topic's rules to one artifact use the meta `review` skill.

Build the Eval Containers image fleet — or any single artifact or eval combination — with `docker buildx bake`, and write the per-artifact `docker-bake.hcl` files the build graph requires. Use when building an image locally, composing a benchmark × agent × model eval, or adding a bake file for a new artifact. For cutting and pushing a tagged release of the whole fleet (with the readiness gate), use the `release` skill, which wraps this one.

Use when adding a new benchmark image to the fleet — a Dockerized task plus verifier that evaluates an agent. Walks the directory layout, the five authored files, the two version knobs, required labels, and the three deployment surfaces. For packaging the AI system that runs *inside* a benchmark (install + entrypoint, LLM access), use add-agent instead.

Cut a release of the Eval Containers image fleet — pass the readiness gate, then build, tag, and push every benchmark, agent, model, gateway, and core image to the registry in bulk. Use when tagging a release or pushing the fleet to ghcr.io/exgentic. For building images during local dev (one artifact at a time, no push, no readiness gate), use the `build` skill instead; this skill is the full release flow that wraps it.

Run the whole-repository health audit that a release manager reads before cutting a tag — run every mechanical gate, reconcile counts against the README, walk the ten fleet questions (does every benchmark build, does every released benchmark have a fixture, do the docs match disk, are agent versions pinned, is CI real), and classify a red / yellow / green verdict. Use this for "audit the fleet", "is the repo release-ready", a quarterly health check, or as step 25 of the `verify` release walk. This is the whole-repo cross-cutting pass; for a single Dockerfile use audit-dockerfile, for a single trajectory use audit-trajectory, and for RULES-text-vs-code drift use audit-rules-drift.

Run a judgment-level health review of a recorded evaluation trajectory in two halves — was the task the agent saw well-formed (the prompt), and was the actual run sane (refusals, retries, wrong answers, API errors, cost). This is the procedural layer that catches content-filter refusals, factually wrong answers on verifiable benchmarks, and tasks delegated to an external file that the mechanical trajectory rule catalog cannot see. Use this for "audit the fixtures/trajectories", when the mechanical catalog flags a yellow, when a new benchmark batch lands, or as step 24 of the `verify` release walk. This is the per-fixture pass; for per-Dockerfile static health use audit-dockerfile, and for the whole-repo sweep use audit-fleet.

Walk the complete release verification of eval-containers end to end — preflight, sanity gates, build/replay/live sweeps, upstream + security scans, the three procedural audits, docs and CI checks, the fleet report, and the tag/publish/announce flow. Use this when cutting a release tag or asked to "run release verification" / "is this release-ready". This is the whole-pipeline driver; for the individual judgment-level passes it invokes, use the narrower audit skills next door (audit-dockerfile, audit-trajectory, audit-fleet, audit-rules-drift).

Use when adding a new agent image to the fleet — the AI system that runs *inside* a benchmark, packaging an install script plus an entrypoint that reads a task and prints an answer. Walks the two-script contract, the single LLM base-URL, the two version knobs, required labels, and the smoke/replay tests. For building the task-plus-verifier environment the agent runs against, use add-benchmark instead.

Run a judgment-level health review of one or more benchmark/agent Dockerfiles — the procedural layer that catches what the mechanical Dockerfile rule catalog cannot: unreasonable install order, dead code, image bloat, labels that misdescribe the image, unsafe entrypoints, and subtle smells a reviewer would flag on sight. Use this for "audit the Dockerfiles", when the mechanical catalog flags a yellow, when a new benchmark/agent batch lands, or as step 23 of the `verify` release walk. This is the per-file Dockerfile pass; for per-fixture trajectories use audit-trajectory, and for the whole-repo sweep use audit-fleet.

Produce or refresh one benchmark's AUDIT.md — the standing record of what has actually been checked about it (validity, safety, size, speed, cost, distribution). Use it when adding a benchmark, after changing its Dockerfile or grader, when its AUDIT.md is stale, or as the per-benchmark step of a fleet audit. It fills the machine-measurable fields from real tool runs and leaves human-judgment fields (running, traces-reviewed, replicate-official, cost) explicitly unchecked until a human does them. For the whole-repo sweep use audit-fleet; this skill drives the oracle (core/oracle/README.md) for the grading-soundness rung.

Generate or refresh containers/AUDIT.md — the project-level table that shows the bottom line of every benchmark's audit in one grid. Use it after any benchmark's AUDIT.md changes, or to see fleet-wide audit status at a glance. It scrapes each benchmarks/<name>/AUDIT.md and copies status verbatim, never inventing status the source files don't carry. For one benchmark's report use audit-benchmark.