inspect-eval-execution

name	inspect-eval-execution
description	Guide for running SABER inspect_ai evaluations locally. Use this when asked to run, re-run, or configure an inspect eval for any SABER domain.

To run SABER inspect_ai evaluations locally, follow this process:

1. Understand the Domain Structure

Each domain lives in domains/<domain>/ and follows this layout:

domains/<domain>/
  <domain>.py          # @task entry point — calls create_task()
  eval.yaml            # Domain metadata, Docker image definitions
  tasks/
    global.yaml        # Default prompts, tools, max_steps, aggregation
    <task_group>/
      <task>.yaml      # Individual task definitions
  compose/
    sandbox.compose.yml  # Docker Compose for sandbox containers
  scoring/             # Domain-specific scoring strategies
  tools/               # Domain-specific MCP tools
  prompts/             # Jinja2 prompt templates (instructions/, assistants/, etc.)
  docker/              # Dockerfiles for sandbox images

The <domain>.py file is minimal — it just calls saber.task.create_task(**kwargs) which wires up config loading, prompt rendering, scoring, tools, and the agent.

2. Build Docker Images First

Before running evals, ensure Docker images are built:

# Build all images for a domain
uv run saber build <domain>

# Or auto-build during eval (slower, builds missing images)
uv run inspect eval domains/<domain> --model <model> --display plain -T build=true

# Force rebuild all images
uv run inspect eval domains/<domain> --model <model> --display plain -T rebuild_all=true

# Rebuild specific images by prefix
uv run inspect eval domains/<domain> --model <model> --display plain -T rebuild=sandbox

3. Run an Evaluation

Always use --display plain to disable the Rich interactive progress display. This outputs clean, parseable text instead of ANSI escape sequences and cursor manipulation that pollute AI agent context windows.

# Basic evaluation (all tasks)
uv run inspect eval domains/<domain> --model <model> --display plain

# Common model formats (we own these endpoints):
#   openai/azure/gpt-4.1        (Azure OpenAI)
#   openai/azure/gpt-5.2        (Azure OpenAI)

# Run a single sample for quick iteration
uv run inspect eval domains/<domain> --model <model> --display plain --limit 1

# --limit <> is the BEST way to keep your evals small and contained for testing

# Filter to specific tasks
uv run inspect eval domains/<domain> --model <model> --display plain -T task_filter="sanity_*"
uv run inspect eval domains/<domain> --model <model> --display plain -T task_filter="incident_response_01"

# Multiple task patterns (comma-separated)
uv run inspect eval domains/<domain> --model <model> --display plain -T task_filter="sanity_*,advanced_*"

3.1 Permanent Environments

It is generally recommended if you expect to do multiround eval analysis to use "-T keep_permanent=true" in your uv run inspect command. This keeps long to spin up resources like databases around for ease of use on following evals.

If you do this, use

uv run saber teardown -y

To teardown any assets when you are finished

4. Key -T Parameters

Parameter	Purpose	Example
task_filter	Glob pattern to select tasks	-T task_filter="sanity_*"
build=true	Build missing Docker images	-T build=true
rebuild_all=true	Rebuild ALL Docker images	-T rebuild_all=true
rebuild=<prefix>	Rebuild images matching prefix	-T rebuild=sandbox
agent=<name>	Agent implementation	-T agent=react
run_preflight=true	Health check before eval	-T run_preflight=true
keep_permanent=true	Keep permanent services alive	-T keep_permanent=true

5. Control Concurrency and Limits

# Limit number of concurrent samples
uv run inspect eval domains/<domain> --model <model> --display plain --max-samples 4

# Limit concurrent model API connections
uv run inspect eval domains/<domain> --model <model> --display plain --max-connections 20

# Combine limits for stability
uv run inspect eval domains/<domain> --model <model> --display plain --limit 1 --max-samples 1

5b. Reasoning models — set --reasoning-effort explicitly (IMPORTANT)

Reasoning models (gpt-5.x, o1/o3, Claude Opus) do NO reasoning by default in these runs: reasoning_effort is unset, the model emits 0 reasoning tokens, and scores land noticeably lower. Always pass it explicitly — especially for cross-run / cross-model comparisons:

uv run inspect eval domains/<domain> --model <model> --reasoning-effort high --display plain

Verified impact (excytin, gpt-5.4, latest_test_set, 599 samples): reasoning=high scored 0.886 vs 0.813 with reasoning unset — a +0.073 aggregate swing, entirely from reasoning, with the largest lift on intermediate checkpoints (the multi-step investigation). The effort used is recorded in the eval-log header (model_generate_config.reasoning_effort); stats.model_usage.*.reasoning_tokens == 0 means no reasoning happened. --max-tokens must be large enough to cover reasoning + answer or completions get truncated.

6. Enable Verbose Logging

# Show info-level logs during eval
INSPECT_LOG_LEVEL=info uv run inspect eval domains/<domain> --model <model> --display plain

# Show debug-level logs (very verbose)
INSPECT_LOG_LEVEL=debug uv run inspect eval domains/<domain> --model <model> --display plain

# Write Python logger output to file
INSPECT_PY_LOGGER_FILE=/tmp/eval_debug.log uv run inspect eval domains/<domain> --model <model> --display plain

7. Iterative Development Workflow

When developing or debugging a domain:

1. Start with --limit 1 to run a single sample quickly
2. Use task_filter to isolate the specific task you're working on
3. Pipe output to capture both stdout and stderr: 2>&1 | tee /tmp/eval_output.log
4. Check the eval log — the path is printed at the end: Log: logs/<timestamp>_<domain>_<id>.eval
5. Iterate on scoring/tools — changes to domain code in domains/<domain>/ take effect immediately (no reinstall needed)
6. Changes to saber or inspect_ai — if installed as editable, take effect immediately; if installed from git, you must edit the files in .venv/lib/python3.11/site-packages/ directly and clear __pycache__ dirs

8. Understand the Score Output

The eval summary shows scores organized by task and scorer:

saber_overall         task_name_1            task_name_2
mean           0.500  accuracy        1.000  accuracy        0.000
stderr         0.000  stderr          0.000  stderr          0.000

• saber_overall is the aggregate across all tasks
• Each task shows its accuracy (the final aggregated score)
• nan means the scorer did not produce a result (e.g., the agent didn't reach that evaluation point)
• Multiple scorers per task (e.g., submission, crash_analysis, root_cause) are aggregated per the task's scoring_aggregation config

9. Common Issues

• "No tasks found": Check task_filter matches task YAML filenames (without .yaml extension)
• Docker build failures: Run uv run saber build <domain> separately to see full build output
• Model API errors: Verify environment variables (AZURE_OPENAI_API_KEY, OPENAI_API_KEY, etc.)
• Stale saber version: uv.lock pins an exact saber commit. If the ACES/SABER library repo has been updated but evals show missing features or unexpected errors, update the lock:

  uv lock --upgrade-package saber && uv sync --all-extras
  

  Then commit the updated uv.lock. Use uv pip show saber to verify the installed commit.