debug-stuck-eval

name	debug-stuck-eval
description	Debug stuck Hawk/Inspect AI evaluations. Use when user mentions "stuck eval", "eval not progressing", "eval hanging", "samples not completing", "eval set frozen", "runner stuck", "500 errors in eval", "retry loop", "eval timeout", or asks why an evaluation isn't finishing.

Quick Checklist

1. Verify auth: hawk auth access-token > /dev/null || echo "Run 'hawk login' first"
2. Get eval-set-id from user (or omit it — most commands fall back to the last ID you used; note watch/status/logs track the last job while list samples/transcript track the last eval set, so the two can differ if you ran a scan after an eval set).
3. Triage with hawk watch — this is the fastest first look. It surfaces per-task / per-sample phase, retries, limits, scores, and pod/scheduling trouble in one snapshot:

   hawk watch <eval-set-id> --no-follow   # one-shot snapshot
   hawk watch <eval-set-id> --json        # raw EvalSetStatus payload for parsing
   hawk watch <eval-set-id>               # live-streaming dashboard (Ctrl-C to stop)
   

4. Interpret the snapshot (see "Reading hawk watch" below) — it usually tells you why without touching logs.
5. Read the trace log: hawk trace <eval-set-id> shows the runner's in-flight actions (an enter with no matching exit = currently hanging). Start with the default tail (-n); don't print --full --raw — the full raw trace can be huge. When you need the whole thing, redirect to a file and let inspect summarize it: hawk trace <id> --full --raw > trace.log then inspect trace anomalies trace.log (--raw keeps the JSON-lines format inspect expects). Only works while the runner pod is running; add -f to follow live.
6. Capture a stack trace: hawk stacktrace <eval-set-id> — py-spy dump of the runner's live thread stacks; shows exactly where the runner is blocked right now. Only works while the runner pod is running.
7. Drill in further as needed: hawk logs <id> for runner output, hawk list samples <id> for the full sample list, hawk status <id> for the raw monitoring report (pod status + metrics + logs as JSON).
8. Test the API directly if logs show retries without a clear error.

Reading hawk watch

hawk watch reads the /monitoring/jobs/{id}/eval-set-status endpoint — the same per-task/per-sample data the web viewer shows. It replaces most manual log-grepping and buffer downloads for triage.

Top-line job status: Queued / Running / Importing samples / Stuck / Complete / Failed. Stuck means Kubernetes reported a trouble reason while pods are still active (it's not progressing on its own).

⚠ trouble_reason (eval-set-level or per-task) — the most actionable signal:

Trouble reason	Meaning	Resolution
runner out of memory (OOMKilled, runner pod)	Runner exhausted memory	Increase runner memory limit
image pull failing (ImagePullBackOff / ErrImagePull)	Bad/inaccessible runner or sandbox image	Check image tag/registry auth
container crash looping (CrashLoopBackOff)	Runner container keeps dying	Check hawk logs for the crash
pods can't be scheduled (Unschedulable)	No K8s capacity past grace period	Wait for autoscale, or check cluster capacity

OOMKilled/CrashLoopBackOff on a sandbox pod is per-sample — Inspect retries it automatically, no action needed. Only runner-pod trouble is job-level.

waiting for Kubernetes — N sandbox pods not yet scheduled — samples whose sandbox pods are still Pending. They aren't in the Inspect buffer yet. A few briefly is normal; many stuck for a while = cluster capacity or image-pull trouble (the reason/detail is in the --json waiting_samples).

Per-sample lines (notable samples: running, errored, hit a limit, or being retried):

• phase: waiting / init / running / scoring / completed / errored / limit / canceled
• N retries — Inspect is re-running this sample. A few is fine; a sample stuck retrying is the thing to investigate (drill into its error/logs).
• limit: <token|time|message|...> — sample halted on a limit, not a bug. Raise the limit if unexpected.
• score <x> — latest score once scored.
• --json adds per-sample last_events (the in-flight event has pending: true) — tells you what a running sample is doing right now (waiting on a model call, a tool, the sandbox), and error, total_tokens, timings.

(status degraded: ...) — a data source (pod_status / metrics / tasks) failed but the rest is usable; not the eval's problem.

Per-Sample Execution Timeline

When one sample is slow or stuck and you need to know where its time went, query the per-sample timeline (HWK-27). It returns the X-Ray span waterfall the runner/middleman already emit — model calls, tool calls, middleman routes, provider requests, scoring, setup — so a single long span (e.g. a hung tool call or provider request) stands out.

No CLI command yet (the web view is a later phase), so curl the endpoint:

TOKEN=$(hawk auth access-token)
curl -s -H "Authorization: Bearer $TOKEN" \
  https://api.inspect-ai.internal.metr.org/meta/samples/<sample-uuid>/timeline \
  | jq '.spans | sort_by(-.duration_ms) | .[0:10] | .[] | {name, category, duration_ms}'

Response: {sample_uuid, eval_set_id, eval_id, spans: [...], truncated}. Each span has span_id, parent_id (build the tree via parent_id → span_id), name, category, kind, start_ms, end_ms, duration_ms, attributes. Categories: setup, run, sample, execution, scoring, model, tool, middleman, provider, heartbeat, other.

Empty spans is not an error — it means tracing is off for that deployment, the eval predates span retention, or it was never traced. The query is on-demand (Logs Insights latency, a few seconds) with no cache.

Error Patterns (in logs / buffer)

Log Pattern	Meaning	Resolution
[uuid task/id/epoch model] Retrying request to /responses	OpenAI SDK retry with sample context	Test API directly with curl to see real error
[uuid task/id/epoch model] -> model retry N ... [ErrorType code]	Inspect retry with error summary	Check error type; use curl for full details
500 - Internal server error	API issue	Download buffer, find failing request, test through middleman AND directly to provider
400 - invalid_request_error	Token/context limit exceeded	Check message count and model context window
Pod UID mismatch	Sandbox pod was killed and restarted	No fix needed—sample errored out, Inspect will retry
Empty output, pending: true	API returned malformed response	Restart eval (buffer resumes)
OOMKilled in pod status	Memory exhaustion	Increase pod memory limits

Key Techniques

1. Start with hawk watch, not logs. The per-sample phase/retry/limit/trouble view answers "why is it stuck" most of the time without grepping. Reach for logs/buffer only when it points at an API error.
2. Retry messages have sample context - All retry messages include a [sample_uuid task/sample_id/epoch model] prefix. Inspect's own retries also include a compact error summary suffix like [RateLimitError 429 rate_limit_exceeded]. The OpenAI SDK's internal retry messages still don't show the actual error — use curl for full details.
3. FAIL-OK patterns are fine - Alternating failures and successes mean the eval IS progressing. Only worry about consistent FAIL-FAIL-FAIL patterns.
4. Use S3 for buffer access - Download .buffer/ from S3 rather than accessing the runner pod directly.
5. Read .eval files with inspect_ai - Use from inspect_ai.log import read_eval_log instead of manually extracting zips.

Test API Directly

Middleman is the auth proxy. If middleman fails but direct provider calls work, it's a middleman issue.

TOKEN=$(hawk auth access-token)

# Test through middleman
curl --max-time 300 -X POST https://middleman.internal.metr.org/anthropic/v1/messages \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"model": "claude-sonnet-4-20250514", "max_tokens": 100, "messages": [{"role": "user", "content": "Say hello"}]}'

# Test OpenAI-compatible
curl --max-time 300 -X POST https://middleman.internal.metr.org/openai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 100}'

Recovery

hawk stop <eval-set-id>      # graceful: score active samples' partial work, then exit
hawk delete <eval-set-id>    # hard: kill the job and tear down K8s resources
hawk eval-set <config.yaml>  # restart — resumes from the S3 buffer

The sample buffer in S3 allows Inspect to resume from where it left off (unless you use --no-resume).

HTTP Retry Count

Task progress logs include "HTTP retries: X". High retry counts indicate API instability even while tasks complete.

Severity: Retry count × wait time = stuck duration. E.g., 45 retries × 1800s = 22+ hours stuck.

More Details

See docs/contributing/debugging.md (repo root) for:

• Sample buffer SQL queries
• Detailed API testing examples
• Escalation checklist

References

• Inspect AI Model Providers - Model configuration
• Inspect AI Eval Logs - .eval file format

Filing Issues

• Middleman: https://github.com/metr-middleman/middleman-server/issues
• Hawk: Linear issue on Evals Execution team
• Inspect AI: https://github.com/UKGovernmentBEIS/inspect_ai/issues