inspect-eval-debugging

name	inspect-eval-debugging
description	Guide for debugging inspect_ai evaluation failures, score issues, and model behavior. Use this when eval results are unexpected, scores are wrong, scoring fails, or model output appears corrupted.

To debug inspect_ai evaluation issues in SABER, follow this systematic process:

1. Triage: Identify the Failure Category

Check the eval summary output. Common categories:

Symptom	Likely Cause	Section
Score is nan for all scorers	Agent never reached scoring (sandbox error, tool failure)	§2
Score is 0.0 but expected higher	Scorer ran but agent's work was incorrect or scorer has a bug	§3
Score differs across runs of same model	Non-deterministic model behavior	§4
Eval crashes with traceback	Code error in domain scoring/tools or inspect_ai	§5
0/0 trailing brace or garbage in tool args	Model produces malformed tool call arguments	§4
Docker/sandbox errors	Container issues	§6

2. Debug Agent Behavior (Score is nan or Unexpected)

Parse the eval log to see exactly what the agent did:

import json, zipfile

with zipfile.ZipFile("logs/<file>.eval", "r") as zf:
    for name in zf.namelist():
        if not name.startswith("samples/"):
            continue
        sample = json.loads(zf.read(name))

        print(f"Task: {sample.get('id')}")
        print(f"Scores: {sample.get('scores', {}).keys()}")

        # Walk through the conversation to see agent actions
        for i, msg in enumerate(sample.get("messages", [])):
            role = msg.get("role", "?")
            if role == "assistant":
                # Check for tool calls
                tcs = msg.get("tool_calls", [])
                if tcs:
                    for tc in tcs:
                        fn = tc.get("function", "?")
                        args = tc.get("arguments", {})
                        cmd = args.get("cmd", "") if isinstance(args, dict) else str(args)[:100]
                        print(f"  [{i}] TOOL: {fn}({cmd[:80]})")
                else:
                    content = msg.get("content", "")
                    text = content if isinstance(content, str) else str(content)[:100]
                    print(f"  [{i}] ASSISTANT: {text[:100]}")
            elif role == "tool":
                text = msg.get("content", "")[:80]
                print(f"  [{i}] TOOL_RESULT: {text}")

3. Debug Scorer Issues (Score is 0.0 Unexpectedly)

3a. Check the score explanation

import json, zipfile

with zipfile.ZipFile("logs/<file>.eval", "r") as zf:
    for name in zf.namelist():
        if not name.startswith("samples/"):
            continue
        sample = json.loads(zf.read(name))
        for scorer_name, score_data in sample.get("scores", {}).items():
            print(f"--- {scorer_name} ---")
            print(f"  value: {score_data.get('value')}")
            print(f"  explanation: {score_data.get('explanation', 'none')[:500]}")
            metadata = score_data.get("metadata", {})
            if metadata:
                print(f"  metadata: {json.dumps(metadata, indent=2)[:300]}")

3b. Test the scorer in isolation

Write a standalone test or script that calls the scorer directly against a known-good input. For example, to test a patch-verify scorer:

# Enter the sandbox container manually
docker exec -it <container_id> bash

# Verify the patch applies
cd /workspace/source
patch -p1 --dry-run < /submit/patches/patch.diff

# Run the build
bash build.sh

# Run POVs manually
./build/fuzz_harness /workspace/povs/pov_0.blob

3c. Add debug logging to scorer code

Domain scoring code lives in domains/<domain>/scoring/. Add logger.info() or logger.warning() calls and run with INSPECT_LOG_LEVEL=info:

INSPECT_LOG_LEVEL=info uv run inspect eval domains/<domain> --model <model> --display plain --limit 1

Domain loggers use the saber.domains.* namespace and inherit inspect_ai's log handling automatically.

4. Debug Model Behavior (Non-Deterministic Issues)

4a. Scan multiple eval logs for patterns

import json, zipfile, os

log_dir = "logs"
for f in sorted(os.listdir(log_dir)):
    if not f.endswith(".eval") or "<domain>" not in f:
        continue
    issues = []
    total = 0
    with zipfile.ZipFile(f"{log_dir}/{f}", "r") as zf:
        for name in zf.namelist():
            if not name.startswith("samples/"):
                continue
            sample = json.loads(zf.read(name))
            for msg in sample.get("messages", []):
                if not isinstance(msg, dict):
                    continue
                for tc in msg.get("tool_calls", []):
                    total += 1
                    args = tc.get("arguments", {})
                    if isinstance(args, dict):
                        cmd = str(args.get("cmd", ""))
                        # Check for trailing garbage brace
                        if cmd.rstrip().endswith("}"):
                            issues.append(cmd[-:])
     issues:
        ()
         ex  issues[:]:
            ()

4b. Known model quirks

• gpt-5.2 trailing }: Sporadically appends } to bash commands in tool call arguments. This is model-generated garbage — it appears in the raw API response from the OpenAI SDK, before any inspect_ai processing. Frequency varies: 0-26% of tool calls per run.
• gpt-5 series uses Responses API: openai.responses.create() instead of client.chat.completions.create(). The code path in inspect_ai is completely different — see §5.
• Reasoning tokens: Models like o1/o3 and gpt-5 produce reasoning tokens that don't appear in visible content but consume output token budget.
• Unset reasoning_effort silently disables reasoning (check this FIRST for score drops): if the same model/dataset scores differently between runs, compare model_generate_config.reasoning_effort and stats.model_usage.*.reasoning_tokens in each .eval header. Unset ⇒ 0 reasoning tokens ⇒ lower scores. Verified: excytin gpt-5.4 latest_test_set scored 0.813 (unset) vs 0.886 (--reasoning-effort high), a +0.073 swing — larger than most scoring/judge changes. Rule this out before blaming data, scoring, or judge-prompt changes.

5. Debug inspect_ai Code Path Issues

5a. Determine which API path a model uses

# Check in the provider code:
# external/inspect_ai/src/inspect_ai/model/_providers/openai.py

# gpt-5 series: responses_api = True (automatic)
# o-series (except o1-early): responses_api = True
# codex models: responses_api = True
# Everything else: Completions API (default)

Key code paths:

API	Entry Point	Tool Call Processing
Completions	OpenAICompatibleAPI.generate() → _generate_completion()	chat_tool_calls_from_openai() in _openai.py
Responses	OpenAIAPI.generate() → generate_responses()	openai_responses_chat_choices() in _openai_responses.py

5b. Find which file Python actually imports

Critical: inspect_ai may be installed from a git URL, NOT as an editable install from external/. Editing files in external/inspect_ai/src/ may have NO EFFECT:

# Check which file is loaded
uv run python -c "import inspect_ai.model._providers.openai_responses as m; print(m.__file__)"

# If it prints .venv/lib/.../site-packages/..., edit THAT file, not external/

5c. Add debug logging to inspect_ai internals

Since inspect_ai uses a custom LogHandler that can suppress standard logger output, the most reliable debugging approach is to write directly to a file:

# Add this temporarily at the injection point:
with open("/tmp/debug_output.log", "a") as _dbg:
    _dbg.write(f"DEBUG: variable={value!r}\n")
    _dbg.flush()

Do NOT rely on logger.warning() or sys.stderr.write() — inspect_ai's Rich console and custom LogHandler can swallow these. File I/O always works.

After editing files in .venv/lib/.../site-packages/, always clear bytecode caches:

find .venv/lib/python3.11/site-packages/inspect_ai/model -name '__pycache__' -exec rm -rf {} +

5d. Using INSPECT_PY_LOGGER_FILE

An alternative to file-based debug prints:

INSPECT_PY_LOGGER_FILE=/tmp/inspect_debug.log uv run inspect eval domains/<domain> --model <model> --display plain --limit 1

This captures Python logger output to a file, but only for loggers in the inspect_ai namespace.

6. Debug Docker/Sandbox Issues

6a. Check container status during eval

# List running containers
docker ps --filter "name=saber"

# Check container logs
docker logs <container_id> 2>&1 | tail -50

# Enter container for manual inspection
docker exec -it <container_id> bash

6b. Verify sandbox compose file

# Validate compose syntax
docker compose -f domains/<domain>/compose/sandbox.compose.yml config

# Check environment variables are set
docker compose -f domains/<domain>/compose/sandbox.compose.yml config | grep -A5 environment

6c. Rebuild images from scratch

# Full rebuild (ignoring cache)
docker build --no-cache -t saber/<domain>/sandbox:latest -f domains/<domain>/docker/Dockerfile.sandbox domains/<domain>/

# Or via saber CLI
uv run saber build <domain> --rebuild

7. Comparing Results Across Models

When the same task produces different scores on different models, use this workflow:

1. Run both models on the same task with --limit 1:

   uv run inspect eval domains/<domain> --model openai/azure/gpt-4.1 --display plain --limit 1 -T task_filter="<task>"
   uv run inspect eval domains/<domain> --model openai/azure/gpt-5.2 --display plain --limit 1 -T task_filter="<task>"
   

2. Extract and compare tool call sequences from both eval logs (see §2)

3. Diff the agent strategies: Look at which commands each model ran, in what order, and where they diverged

4. Check for model-specific issues: Some models produce different JSON formatting, escape sequences, or garbage characters in tool call arguments

8. Rapid Iteration Checklist

When debugging a failing eval:

1. Run with --limit 1 and task_filter to isolate
2. Check the score explanation in the .eval ZIP
3. Walk the agent's tool call sequence
4. If scorer issue: add logging to domains/<domain>/scoring/, run with INSPECT_LOG_LEVEL=info
5. If model issue: scan multiple runs for the pattern
6. If inspect_ai issue: find the actual imported file (__file__), add file-based debug logging, clear __pycache__
7. If Docker issue: check container status, enter container manually
8. After fixing: re-run eval, verify score changed as expected
9. Clean up all debug logging before committing