القائمة
Diagnose and fix failed benchmark runs. Use when a benchmark run fails, scores are unexpected, judges show errors, or traces are incomplete.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
| name | debug-benchmark |
| description | Diagnose and fix failed benchmark runs. Use when a benchmark run fails, scores are unexpected, judges show errors, or traces are incomplete. |
Diagnose failures across the benchmark execution chain.
CLI args → packages/cli/src/args.ts
↓
Runner setup → packages/runner/src/index.ts
↓
Agent lifecycle → packages/runner/src/agent-lifecycle.ts ← START HERE for agent failures
↓
Adapter exec → packages/adapters/src/<adapter>-adapter.ts
↓
Judge eval → packages/judges/src/index.ts ← START HERE for judge failures
↓
Report gen → packages/report/src/scoring.ts
↓
Output → packages/report/src/decision-report.ts
| Check | File | What to look for |
|---|---|---|
| CLI installed? | agentarena doctor | Check PATH |
| Adapter registered? | packages/adapters/src/adapter-registry.ts | Import + entry in adapterEntries |
| Preflight passes? | <adapter>-adapter.ts → preflight() | Returns ready or blocked with reason |
| Binary override? | Environment AGENTARENA_<AGENT>_BIN | Custom path support |
The runner lifecycle at packages/runner/src/agent-lifecycle.ts (837 lines) has these phases:
prepareWorkspace() → setupSandbox() → executeAgent() → collectResults() → evaluateJudges()
| Phase | File | Debug tip |
|---|---|---|
| prepareWorkspace | agent-lifecycle.ts | Check sandbox directory was created |
| setupSandbox | agent-lifecycle.ts | Check repo was copied to sandbox |
| executeAgent | agent-lifecycle.ts | Check agent process ran and produced output |
| collectResults | agent-lifecycle.ts | Check stdout/stderr were captured |
| evaluateJudges | agent-lifecycle.ts | Check judge results exist |
Add console.log or inspect trace at .agentarena/runs/<run-id>/trace/.
| Symptom | File | Likely cause |
|---|---|---|
command judge fails | packages/judges/src/command-runner.ts | Command not found or timeout |
file-exists fails | packages/judges/src/judges/ | Working directory wrong |
lint-check fails | packages/judges/src/shared.ts | Linter not configured for the repo |
| All judges critical fail | packages/judges/src/index.ts → runJudge() | Agent produced no output at all |
Command runner at packages/judges/src/command-runner.ts handles process execution with:
resolveJudgeWorkingDirectory())| Check | File |
|---|---|
| Weight presets sum to 1.0? | packages/report/src/scoring.ts |
| Token budget parsed? | packages/report/src/scoring.ts → computeEfficiencyScore() |
| Mode matches CLI? | packages/cli/src/args.ts vs scoring modes in scoring.ts |
| Issue | File |
|---|---|
| HTML report wrong | packages/report/src/html-template.ts (502 lines) |
| Decision unclear | packages/report/src/decision-report.ts (383 lines) |
| Leaderboard missing | packages/report/src/leaderboard.ts (311 lines) |
| Markdown wrong | packages/report/src/markdown-template.ts (277 lines) |
Trace files live in .agentarena/runs/<run-id>/trace/:
# List traces
ls .agentarena/runs/
# Read trace events
cat .agentarena/runs/<run-id>/trace/events.json
# Replay a trace programmatically
# See packages/trace/src/replay.ts (406 lines)
# Check agent availability
node packages/cli/dist/index.js doctor --probe-auth
# Run a single quick benchmark
node packages/cli/dist/index.js run --repo . --task examples/taskpacks/demo-repo-health.json --agents demo-fast
# Run with JSON output for inspection
node packages/cli/dist/index.js run --repo . --task examples/taskpacks/demo-repo-health.json --agents demo-fast --json
# Check trace
node --test tests/trace.test.mjs
pnpm build && pnpm test passes after the fix.--agents demo-fast).Plan and execute code changes using a spec-driven workflow with structured documents. Use when making non-trivial changes (multi-file, multi-step, or behavior-impacting) that need structured planning before implementation. Not for trivial single-file edits.
Create and manage changesets for versioning with @changesets/cli. Use when making changes that need a changelog entry, preparing a release, or managing package versions.
Build the pnpm monorepo correctly respecting inter-package dependencies. Use when building, verifying changes, or troubleshooting build failures.
Write and run tests using Node.js built-in test runner (node:test) with .mjs files. Use when creating new tests, debugging test failures, or understanding test infrastructure.
Understand the vanilla JS web-report app structure, patterns, and contribution guidelines. Use when modifying the UI, adding new views, or fixing frontend bugs.
Add a new AI coding agent adapter so it can participate in benchmarks. Use when integrating any new CLI-based coding tool (Cursor-like, Copilot alternative, open-source agent, etc.) into the adapter system.