model-benchmark

name

model-benchmark

description

Create, inspect, resume, analyze, integrity-verify, and seal reusable Harbor/Codex model benchmark campaigns. Use when an agent in Codex CLI, Claude Code, Gajae Code, or Hermes Agent needs to compare exact model IDs on a repository-editing dataset; reuse an existing model-benchmark workspace; run model, container-auth, and oracle gates; prevent network or oracle leakage; aggregate pass@1, latency, token, and cost results; or audit preserved benchmark artifacts.

Model Benchmark

Build every comparison as a new immutable campaign. Reuse prepared datasets and vendor sources, but never overwrite an existing raw job or campaign.

Resolve the runner

Resolve the absolute directory containing this loaded SKILL.md and call it SKILL_ROOT. Do not assume the current working directory is the skill directory. Then use:

CTL="$SKILL_ROOT/scripts/benchmarkctl.py"
python3 "$CTL" --version

Read references/platforms.md only when the client does not expose the loaded skill path or when installing the skill. Read references/workflow.md when adapting datasets, diagnosing a gate, or explaining the methodology. Read references/workspace.md when a Harbor workspace, its vendored sources, or its datasets are missing and must be rebuilt from pinned references.

Check prerequisites

Require:

• Python 3.9 or newer;
• a prepared Harbor workspace with scored and reference-solution oracle datasets;
• Docker and a working Harbor CLI;
• a pinned Codex CLI and readable Codex auth file;
• exact model IDs and explicit permission before live model calls.
• when a workspace is absent, rebuild it from pinned references with scripts/bootstrap_workspace.py and workspace.lock.json; see references/workspace.md.

The orchestration client may be Codex CLI, Claude Code, Gajae Code, or Hermes Agent. The benchmark runner itself intentionally invokes the Codex CLI through Harbor so the compared execution surface remains consistent.

Preserve integrity

• Compare exact model IDs, not mutable aliases.
• Freeze source revisions, input tree hashes, task lists, CLI version, bundled policy, bundled analyzer, and resolved configs before model calls.
• Require one attempt, zero retries, disabled web search, per-model concurrency one, and global concurrency from one through four.
• Run host model preflight, a Harbor container auth smoke, and all reference-solution oracle jobs before scoring.
• Require oracle task coverage to equal the scored task set.
• Require scored Docker recipes to remove packaged .oracle data.
• Preserve raw logs. Treat missing rewards, dependency failures, cancellations, and harness exceptions as infrastructure errors rather than model failures.
• Analyze only a complete task-by-model grid and require all analyzer audits to pass.
• Seal the verified campaign with SHA-256 and treat a sealed campaign as read-only.

Protect active work

Before creating a campaign, inspect the workspace for running or recently modified jobs. If another session is active:

1. Read its source inputs only.
2. Create a distinct campaigns/<unique-name> directory or use --campaign-root outside the workspace.
3. Do not edit its configs, results, logs, datasets, or reports.
4. Do not start another live benchmark unless the user explicitly requested the model-call and Docker cost.

Create and check a campaign

Locate the workspace, scored dataset, oracle dataset or datasets, Harbor executable, Codex executable, auth file, and exact model IDs. The bundled analyzer, network overlay, and policy are the defaults; override them only for a documented reason.

python3 "$CTL" init \
  --workspace /path/to/model-benchmark \
  --name exact-model-a-vs-b \
  --model exact-model-id-a \
  --model exact-model-id-b \
  --dataset datasets/scored-suite \
  --oracle-dataset datasets/oracle-suite

init prints the new benchmark-plan.json path. Store it in PLAN. It fails instead of overwriting an existing campaign. Repeat --oracle-dataset when oracle coverage is split.

Validate frozen inputs and render the whole pipeline without model calls:

python3 "$CTL" check --plan "$PLAN"
python3 "$CTL" pipeline --plan "$PLAN"

Do not continue when check reports ok: false. Correct the source workspace and create a new campaign rather than mutating a frozen plan.

Execute only when authorized

After an explicit request to spend model calls and run Docker jobs:

python3 "$CTL" pipeline --plan "$PLAN" --execute

The pipeline runs static checks, exact-model preflight, per-model Harbor auth smoke, oracle jobs, the scored grid, analysis, verification, and artifact sealing. It skips stages already proven complete. If an existing stage directory is incomplete or invalid, preserve it and create a newly named campaign.

Resume or operate one stage

Run status first in a later session. A valid seal is the terminal state.

python3 "$CTL" status     --plan "$PLAN"
python3 "$CTL" preflight  --plan "$PLAN" [--dry-run]
python3 "$CTL" auth-smoke --plan "$PLAN" [--dry-run]
python3 "$CTL" oracle     --plan "$PLAN" [--dry-run]
python3 "$CTL" run        --plan "$PLAN" [--dry-run]
python3 "$CTL" analyze    --plan "$PLAN"
python3 "$CTL" verify     --plan "$PLAN"
python3 "$CTL" seal       --plan "$PLAN"

Complete with evidence

Finish only when:

• static input and config checks pass;
• every exact model passes host preflight and Harbor auth smoke;
• every oracle task returns reward 1 without errors;
• the scored job contains the exact task-by-model grid;
• binary rewards are complete, unresolved errors and policy violations are zero, and concurrency audits pass;
• analysis/summary.json reports audits.official_run_valid: true;
• verification.json reports ok: true;
• artifacts.sha256 exists and status reports a valid seal.

Report the campaign path, exact models, task and trial counts, completed gates, result summary, and limitations. Describe a one-attempt small sample as an observed benchmark run, not a general leaderboard result.