메뉴
Compare Claude Code and Codex on the same real code-change task with isolated worktrees, identical gates, transcripts, time, and cost.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
Align AGENTS.md, rules, README, plans, and chronicles with the codebase; use --clean to archive obsolete task docs into ATLAS.md.
Research current best practices and turn the evidence into a clear recommendation. Use for state-of-the-art, evidence-based, optimal-approach, comparison, or pros-and-cons questions in any field.
Clarify an ambiguous or consequential change before planning or implementation when more than one sound approach exists.
Add CHANGELOG.md entries, derive them from commits, or cut a Keep a Changelog and SemVer release.
Create a user-requested Conventional Commit from the staged changes.
Define or implement regression proof for a project or business flow. Use for test strategy, black-box or deep-integration tests, business KPIs and thresholds, test audits, missing tests, or test implementation.
SOC 직업 분류 기준
| name | ai-agent-bench |
| description | Compare Claude Code and Codex on the same real code-change task with isolated worktrees, identical gates, transcripts, time, and cost. |
| user-invocable | true |
| allowed-tools | Glob, Grep, Read, Bash, Edit, Write |
Benchmark agents only when the task, starting commit, and outcome check are identical. The harness keeps each result branch and removes its temporary worktree.
Create <repo>/.agent-bench.toml:
prompt = "prompts/task.md"
start_branch = "main" # or start_commit
agents = ["claude", "codex"]
outer_check = "./scripts/full_check.sh"
inner_check = "pytest tests/integration/test_x.py -q"
outer_check is both the before/after gate and wall-time measure; it must exercise the real outcome. inner_check is the fast command agents use while working.
Require a clean repository, available CLIs, and a passing outer_check before trials. Confirm the agents and next run ID, then run each agent sequentially so concurrent load cannot corrupt timing:
python <skill>/scripts/run_trial.py --repo "$REPO" --config "$REPO/.agent-bench.toml" --agent "$AGENT" --run "$RUN_ID"
Results live under eval-results/<task>/<agent>/run-<id>-<timestamp>/. Unexpected harness or agent behavior is appended to ai-agent-bench-anomalies.md; details are in anomalies.md.
Aggregate completed runs with scripts/parse_transcript.py --aggregate <run-dirs> --output comparison.json --render-report comparison.md. Report gate results, branches, time delta, tokens, and cost; never rank a failed trial as faster.
For plugin behavior rather than a real code task, use the Pydantic runner documented by eval-regression and scripts/fresh_context_eval.py.
Never commit on the user's branch. A repeated run creates a new timestamped result and preserves prior evidence.