| name | run-eval |
| description | Run ONE evaluation — a single Task × Model × AgentConfig — end to end, local or on the bastion; invoke when the user wants to run, evaluate, or "kick off" a single eval combo (e.g. "run secret-rotation on gemini-3.1-pro-preview", "evaluate this task once and watch it"). |
Run a single eval
The easy entrypoint for one eval run: one Task × one Model × one AgentConfig.
A single run is just a 1×1×1 matrix, driven through the same wrapper everything
else uses. This skill keeps the front-end lean — no concurrency, no combo math,
no cross-combo safety. For those, use a different skill (see Wrong tool?).
The run mechanics are shared and live in the references — read them, don't
re-derive them here:
Flow
1. Choose where + how, then pin the one combo
- Local or bastion? Local is the default; remote sets
BENCH_REMOTE=1 + the
BASTION_* connection env. Auth: prefer Vertex/ADC (BENCH_VERTEX=1, no
keys). Both covered in running-evals.md.
- Pin exactly one Task (
task.yaml path), one Model, one AgentConfig
(oc|gcli [+mcp][+skills], refactored arm only), and the arm (refactored
run_matrix.sh or legacy run_matrix_legacy.sh, oc-only).
- Ask the operator for anything not given — don't guess on the dimensions that
cost a cluster + time (≈25–40 min for an infra-bearing task).
2. Preview with DRY_RUN=1
Show the expanded (single) combo + per-combo env before spending a cluster:
DRY_RUN=1 \
MATRIX_TASKS="tasks/gcp/secret-rotation/task.yaml" \
MATRIX_MODELS="gemini-3.1-pro-preview" \
MATRIX_AGENT_CONFIGS="gcli+mcp+skills" \
scripts/bastion/run_matrix.sh
3. Clean pre-flight
Work the "Before any retry" checklist in
../../../docs/appendix/known_issues.md
before launching — stale per-run state and orphaned cloud resources are the top
cause of a "fresh" run failing instantly. For leaked clusters / gke-nodes-*
SAs / secrets, use the cleanup-orphaned-resources skill.
4. Launch the one combo (detached)
Drive the wrapper with single-value MATRIX_*. It runs detached under nohup
and prints a STAMP — record RESUME_STAMP=<stamp> in durable state; it is your
handle for monitoring, retry, and re-attach. Example (Vertex, refactored arm;
prefix BENCH_REMOTE=1 + BASTION_* for remote):
BENCH_VERTEX=1 AGENT_PROVIDER=google-vertex \
JUDGE_PROVIDER=google JUDGE_MODEL=gemini-3.1-pro-preview \
MATRIX_TASKS="tasks/gcp/secret-rotation/task.yaml" \
MATRIX_MODELS="gemini-3.1-pro-preview" \
MATRIX_AGENT_CONFIGS="gcli+mcp+skills" \
RESULTS_DIR="results/<label>" \
scripts/bastion/run_matrix.sh
(Legacy arm: run_matrix_legacy.sh, oc-only, no MATRIX_AGENT_CONFIGS.
MAX_PARALLEL is irrelevant for one combo.) Full launch details, including
running both arms, are in running-evals.md.
5. Watch it
Basic loop: poll the combo's status + run.log tail on an interval (~3–5 min
for an infra-bearing task — don't busy-poll). If your poller dies the
detached run continues; re-attach with RESUME_STAMP=<stamp> and the same
command. For a resilient, hands-off watch that survives quiet stretches and dead
watchers, follow
../../references/monitoring-and-recovery.md
(for one combo it degenerates to one watcher + the keepalive loop).
Classify a flake vs a real failure against the router in
known_issues.md: infra flake → clean +
retry (cap 2); real failure (auth/config, low score, task-logic) → do not retry,
analyze.
6. Summarize + where to read scores
When the combo has a terminal status (or .done), report:
task · model · agent-config · arm · auth-mode · exit · score · pass/fail
checks. Results land per the layout in
running-evals.md; for how scoring works and
how to interpret it, see
../../../docs/components/metrics.md. If
it didn't pass, give the decisive log line (redact secrets), the root cause, and
the concrete fix. Then confirm teardown is clean.
Wrong tool?
- Validating a NEW eval (iterate in a loop until the task is correct and
runnable, hands-off or self-healing) → use the
validate-eval skill. No loop logic lives here.
- More than one combo (a matrix, a comparison) → use the
run-parallel-evals skill — it adds combo
math, MAX_PARALLEL, and cross-combo parallel-safety.
Guardrails
- One run still costs a real GKE cluster + ~25–40 min — confirm scale and
DRY_RUN=1 first.
- Never print or commit API keys; redact secrets in summaries.
- Never launch a second run of the same task+model+config concurrently
(run-id-derived cluster names are reused).
- Always confirm clean teardown — orphaned
gke-nodes-* SAs silently 409 the
next run.
- Prefer leaving the run detached + re-attaching via
RESUME_STAMP over a
fragile foreground session.
