validate-eval

name	validate-eval
description	Validate that a NEW eval is correct and runnable by running it and iterating in a loop until green — fixing infra/config/task bugs (not the model's score), with opt-in hands-off and unlimited/self-healing modes; invoke when authoring or vetting a task before setting `validated: true`.

Validate a new eval

Prove a newly authored eval is correct and runnable, iterating until it runs clean. The loop fixes the eval, not the model.

[!IMPORTANT] The goal is task correctness, not making the model pass. A successful validation means: the infra provisions, the agent runs end to end, and the verification/judge produce sane scores. A genuine model miss — a clean trajectory that just scores low on a hard task — is a successful validation, not a failure to fix. Do not edit the task to make a weak model pass.

But first rule out a silent capture failure, which masquerades as a low score. Before reading a low score as a model miss, assert the run's trajectory is non-empty (and tools populated for tool-using tasks). An empty trajectory on a task the agent clearly acted on is INVALID — a harness capture failure (e.g. oc sessions exiting 127 because Node isn't on PATH), not a model miss — and must be fixed and re-run, not recorded. See the empty-traj / exit 127 row in known_issues.md.

This skill is pointer-driven. The run mechanics, monitoring, and self-healing loop all live in shared references — read them, don't restate them:

• Local vs bastion, auth, clean pre-flight, launch, knobs, results → ../../references/running-evals.md
• Monitoring / keepalive / recovery → ../../references/monitoring-and-recovery.md
• Unlimited / self-healing loop → ../../references/unlimited-mode.md
• Failure router → ../../../docs/appendix/known_issues.md
• Reading scores → ../../../docs/components/metrics.md
• Capability → tool mapping → ../../references/harness-capabilities.md

---

Modes (opt-in)

Mode	When	What it adds
Standard	default	You drive the loop directly, one attempt at a time.
Hands-off	"watch it", "don't stop", long runs	Resilient monitoring + keepalive — monitoring-and-recovery.md.
Unlimited / self-healing	"keep going until green", "auto-fix and restart"	Diagnose → fix in a worktree → re-sync → restart, capped — unlimited-mode.md.

Resolve the mode up front; hands-off and unlimited are explicit opt-in.

---

Flow

1. Pre-flight the task spec (optional but recommended)

Before spending a cluster, run the task-review skill (or skim the stack/scripts) to catch obvious authoring bugs statically — schema errors, a non-unique task_id, self-collisions (a fixed $HOME repo it rm -rfs, a shared-VM-SA role it grants then revokes), or a mis-scoped rubric. Cheaper to fix on paper than after a 30-min run.

2. Run it

Use the mechanics in running-evals.md: choose local vs bastion + auth, clean pre-flight (the "Before any retry" checklist in known_issues.md), DRY_RUN=1 preview, then launch the one combo detached and capture RESUME_STAMP. Validation is a single combo per attempt — don't fan out.

3. Loop until green

Monitor per monitoring-and-recovery.md. On each finished or flaked attempt, classify the failure against the router in known_issues.md and take exactly one action (the same decision tree as unlimited-mode.md):

• Retry — infra flake (Vertex 429, ssh exit 255, transient quota): clean per the checklist, re-run. No edit.
• Fix + retry — config / auth / host setup (missing API enablement, Vertex ADC marker, inotify limit, folder-trust): apply the router's documented fix, then retry. Environment fix, not eval-logic.
• Fix the task + retry — a real task/stack/rubric bug the validation surfaced (the thing you're here to catch): fix it in an isolated worktree scoped to that bug, then retry. Log the cycle in durable state.
• Escalate / accept — a genuine model-capability low score (clean trajectory, hard task): this is a successful validation. Record it and stop; do not edit the task to inflate the score.

Honor the STOP conditions from unlimited-mode.md: goal met, attempt cap (≤2–3 per attempt), no-progress (same failure signature twice), or budget exhausted (each restart costs a cluster + ~25–40 min). Hands-off and unlimited behaviors are opt-in; without them, surface a persistent failure rather than looping.

4. Confirm green + recommend validated: true

When the eval runs clean, read the scores (metrics.md) and confirm they look sensible — checks pass/fail for the right reasons, no swallowed scoring error, no empty tokens: {} masking a timeout. Then recommend setting validated: true on the task (the flag gates leaderboard promotion). If a real model miss is the only "failure," say so explicitly — the task is validated.

---

Living known-issues

When you hit a failure mode not already in known_issues.md, append a router row so the next run benefits: symptom → root cause → fix → class. Keep it terse, don't duplicate an existing row, and follow the docs conventions (scope + user-guide, GFM, not journal-like, no model scores). This capture step is part of validation, not optional.

---

Guardrails

• Validate the task, not the model. Never edit a task to make a weak model pass — a genuine miss is a valid result.
• Each attempt costs a real GKE cluster + ~25–40 min — DRY_RUN=1 first, cap attempts (≤2–3), and track budget.
• Make all fixes in an isolated worktree/branch; keep commits scoped and local — surface the diff for review, don't push shared branches unless asked.
• Never print or commit API keys; redact secrets in summaries.
• Always confirm clean teardown — orphaned gke-nodes-* SAs silently 409 the next run.
• Hands-off run: never emit a completion signal until the eval is terminal and summarized; emit a periodic heartbeat instead.
• Don't auto-fix what you can't both name and resolve — task-authoring / rubric judgement calls escalate to a human.