interpreting-eval-harness-evaluations

name	interpreting-eval-harness-evaluations
description	Monitor and interpret evaluation runs of the eval-harness framework. Use when asked to watch, interpret, debug or diagnose an eval run, its logs, its scores or its results.

Overview

This guide covers how to monitor an eval-harness run while it is in progress, interpret the logs and results once it finishes, and diagnose the failures that can occur along the way.

Locate the run

Every run writes to its own directory: <OUTPUT_DIR>/<YYYYMMDD_HHMMSS>_<session_id>/, where OUTPUT_DIR is set by EVAL_HARNESS_OUTPUT_DIR (default output). Inside it:

• session.log - every record from every agent thread, interleaved.
• <agent_type>_<model>[_<effort>].log - one per agent, containing only that agent's records. Characters that cannot appear in a filename (e.g. / in model names) are replaced with _.
• results.json (or results.csv) - written when the session finishes; see the Results File Schema for the field-by-field breakdown.
• diagnostics/<agent>/eval-XX/run-XX/attempt-XX/ - present only for failed attempts when EVAL_HARNESS_CAPTURE_FAILURE_DIAGNOSTICS=true; contains failure metadata and best-effort AgentShell trace/container state.

Read the logs

Log lines are formatted <timestamp> - <level> - <file:line> - <message>. The markers to look for:

• --- arrange phase --- / --- act phase --- / --- score phase --- - phase boundaries.
• [arrange] <line> (likewise [act] and [score]) - stdout/stderr streamed from inside the container, including every print() in the eval's phase methods.
• Run 1/3 - logged when an eval has run_count > 1; each run is a fresh container.
• EVAL_TOTAL_TOKENS=<int> - emitted once per agent-shell call and summed into the token totals.
• EVAL_SCORE=<float> - printed by the eval's score method; the harness takes the last occurrence in the score phase output. If no line matches, the run silently scores 0.0.
• phase <label> completed - the phase finished cleanly.

In the results file, an eval's score is the mean across its runs, while total_tokens and time_taken_seconds are totals across them.

Monitor a live run

tail -f the session log, or a single agent's log for less noise. A failing agent does not stop the others: it is logged as Agent <type>-<model> failed with a traceback, marked FAILED, and the session carries on. At the end, main.py prints one FAILED: ... line per failed agent and exits with code 1.

Diagnose failures

Symptom	Likely cause	Where to look
RuntimeError: ... not configured or ... auth file not found, before any phase marker	Missing or expired credentials for that agent	Authorisation
<phase> failed (exit <N>) followed by a --- container output --- block	The phase script raised; the block holds the full traceback	The traceback; for arrange, also external dependencies (repo clones, MCP servers)
<phase> timed out after <N>s (exit 124 or 137)	The phase exceeded its timeout	EVAL_HARNESS_<ARRANGE|ACT|SCORE>_TIMEOUT_SECONDS, defaults 3600/3600/600
Raw AgentShell details are needed	Failure diagnostics were enabled	The failed attempt's diagnostics/.../failure.json and, when present, agent-shell-trace.log
NameError or ImportError in a phase traceback	The eval violates an authoring constraint (module-level state, imports outside the method body)	Constraints in the eval creation skill
TypeError: <Class> must be a class implementing arrange/act/score	The eval class does not satisfy the EvaluationFile protocol	Same constraints section
Score is 0.0 with no error	The score method never printed EVAL_SCORE= (or the agent genuinely scored zero)	The [score] output for that run
Malformed score line	EVAL_SCORE= was printed with a non-float value	The eval's score method
total_tokens is 0 but the agent clearly did work	Stale docker image (agent-shell version mismatch)	Rebuild the images - see the commands in AGENTS.md
after the end-of-run summary

Where the fix belongs

Classify the failure before changing anything:

• The eval itself (authoring constraint violations, scoring bugs) - fix the eval class, following the eval creation skill.
• Harness configuration (timeouts, output locations) - .env / EVAL_HARNESS_ environment variables, documented in Configuration.
• Credentials - re-authenticate on the host per the authorisation guide.
• The docker image - rebuild manually after any Dockerfile or agent-shell change; there is no auto-rebuild.