| name | eval-agentic-cleanup |
| description | Audit + recover a finished agentic eval. ALWAYS start with the read-only, idempotent completeness/health audit (§0): job finished? score present + non-zero + not obviously broken? HF traces present + linked? trial count ≈ n_rep × benchmark_size? — it writes nothing and recommends an action per check. Then run only the flagged remediations: manual HF-trace upload + Supabase DB-registration via manual_db_eval_push.py, the vLLM-numeric-ID → real-HF-model-name fix (with cross-user FK safety), verify, free disk. Use to verify an eval is truly complete, when a sweep finds an eval that didn't upload/register or has a broken/zero score or short trial count, or to re-register/correct an eval's model/traces. Distinct from the model- publishing cleanups (rl-agentic-job-cleanup / sft-job-cleanup) and datagen-job-cleanup — this is the EVAL path. |
eval-agentic-cleanup
Always start with the read-only audit (§0): it verifies the four conditions that actually matter, then run only the remediations it flags. "The score is in the leaderboard" does NOT mean the eval is complete — the auto-harvest path can land a score while silently skipping trace upload, or record a bogus 0 from an eval that never reached the model.
0. Read-only completeness + health AUDIT — run first (idempotent, ZERO writes)
Reads only (sacct, run-dir result.json, HF API, Supabase reads). Emits a per-eval verdict + recommended action per check; run only the flagged remediations.
Pre-gate — EXEMPT: grid / throughput / OOM-test measurement runs targeting DCAgent2/* → skip (upload only production winners).
| # | Check | How (read-only) | PASS | If it FAILS → recommend |
|---|
| 1 | Job finished | sacct -j <id> -X -o State | COMPLETED | RUNNING/PENDING → wait & re-audit; FAILED/TIMEOUT/CANCELLED → diagnose + relaunch (see eval-agentic-launch gotchas #1–#5: jobs_dir perm / hosted_vllm 2-slash / model_info / reservation / stale-row) |
| 2 | Score present & not broken | Supabase sandbox_jobs score, or run-dir result.json → stats.evals.<k>.metrics[0].mean | non-null real number; 0 OK only if check 4 shows trials ran | null/missing/0 with ~0 trials/POSTs = infra-broken (eval never reached the model — classic jobs_dir/hosted_vllm/model_info failure) → re-run. High VerificationNotCompletedError rate → score may be biased (note below). |
| 3 | HF traces present + linked | trace HF repo exists + non-empty (HF API 200) AND DB/LB row has trace/url field set | repo 200 AND linked | missing/empty OR unlinked → upload traces + register link (§1–§3) |
| 4 | VALID trial count (PARSE, don't file-count) | Parse each per-trial result.json for a numeric reward (snippet below). An errored trial STILL writes result.json with exception_info and no reward, so file-count overstates coverage. | VALID is ~90–100% of n_rep_eval × benchmark_size, with at most n_rep_eval retained attempts per task | VALID or total above plan / more than n_rep_eval attempts for a task → deduplicate (§0a), then re-audit. Short VALID (<~90%, a missing rep, or high hard-error 15–25%) → resume errored trials. Expected: swebench-verified-random-100=100, dev_set_v2=100, terminal_bench_2=89; × n_rep_eval (default 3) → ~300/~300/~267. |
Output one row per eval: ✅/⚠️ per check + the single recommended next action. Re-run the audit after any remediation to confirm ✅.
No-reward trial classes (VNC / infra errors) — bias direction
VerificationNotCompletedError (harbor 9203989f) / pre-fix AgentTimeoutError WITH NO reward = the verifier never produced a result. Missing because the agent burned its budget → true score ≈ 0 (SWE-bench ~pass/fail). Harbor's JobStats mean DROPS these from the denominator → biased UP; the DB/leaderboard harvest counts them as 0 → unbiased. Either way retriable: resume on a healthy sandbox — the payoff is usually recovering real passes hidden as 0s.- Infra errors (
DaytonaAuthenticationError/DaytonaValidationError/etc.) = failure uncorrelated with solvability → dropping is ~unbiased (lowers N). Re-run for completeness only; do NOT impute 0.
AgentTimeoutError WITH a reward = verifier scored it → passthrough VALID 0. Don't filter/re-run.- Discriminate by reward presence, not the error label.
<90% COMPLETE → RESUME, do NOT REGISTER (and DE-REGISTER if it slipped through)
An eval whose VALID-trial count is < ~90% of planned n_rep × benchmark_size (swebench-100=300, tb2=267, dev_set_v2=300) is INCOMPLETE → RESUME to completion (Check-4 resume), NOT register. A partial score silently MIS-RANKS the model. If the auto-harvest already registered a partial (Pending/Started, or a prematurely-Finished partial) → DE-REGISTER that ONE row (cross-user FK-safe per §Remediations); it re-registers cleanly on resume. ≥90% / a ~99% tail-short is fine to finalize. Never run §1's register/upload on a <90% eval. Skip a candidate whose slurm job is still RUNNING.
>10% error-fraction HARD GATE — the authoritative metric
Compute via the ONE utility scripts/database/eval_guardrail.py (guardrail_counts(stats, planned).invalid_error_count), the Python port of the leaderboard guardrail (OT-Agent-Leaderboard server/storage.ts:416-449). It EXCLUDES the benign passthrough error types — never hand-restate the BENIGN set (that's how sibling copies drifted). error-fraction = invalid_error_count / total_attempted; a row is an incomplete partial (→ de-register + resume) iff > 10%. There is NO "borderline"/"close enough" discretion: 10.1% fails like 30%. Do NOT re-derive as a naive "no-reward / valid-reward<90%" count — that folds in the benign passthroughs and over-flags near-complete evals. Exception: an explicit, logged user grandfather of a specific already-pushed row.
Run this gate after deduplication. Duplicate attempts inflate the denominator and can falsely turn a broken run into a pass.
0a. Duplicate-contaminated trials — deduplicate before every later step
An eval is duplicate-contaminated when total trials exceed n_rep_eval × benchmark_size or any task has more than n_rep_eval attempts. Do not upload, score, register, resume, or delete the source run before deduplication.
Use the bundled deterministic script. Run its dry-run first. If after = benchmark_size × n_rep_eval, apply and re-run §0. If fewer unique attempts remain, apply the deduplication, then follow check 4 and resume the missing replicas in the same run:
python scripts/harbor/dedupe_harbor_trials.py \
--job-dir trace_jobs/<RUN_TAG> --n-rep <n_rep_eval>
python scripts/harbor/dedupe_harbor_trials.py \
--job-dir trace_jobs/<RUN_TAG> --n-rep <n_rep_eval> --apply
Selection is score-blind: globally collapse copied paths with the same Harbor trial_name before grouping by task, then keep the earliest numeric-reward attempts and the earliest no-reward attempts only if needed to reach n_rep_eval. Never choose by reward value. This preserves completed original attempts, lets successful infrastructure retries replace unscored attempts, and cannot cherry-pick passes. The reusable scripts/harbor/dedupe_harbor_trials.py utility removes only surplus trial directories, writes dedupe_manifest.json, and rebuilds result.json plus harbor_result.json.
For object-store runs, stage the exact run prefix locally, deduplicate there, verify the parsed matrix and aggregate, then replace only that run's harbor_trials tree and aggregate files. If result files are staged before large trial artifacts, exclude every manifest directory prefix and file path from the original object listing when hydrating retained artifacts; otherwise a stale pre-delete listing can copy a removed nested result back into the local tree. A second invocation must report before == after and remove nothing.
Check 4 — counting VALID trials (parse, never file-count) — STANDARD report element
Trial dirs at eval_jobs/eval-<safe_model>_<safe_dataset>/<task>__<id>/result.json (depth-1; the run-dir-root result.json is the aggregate — exclude via the */ glob). A trial is VALID iff reward is finite; else ERRORED. One pass over all evals (otagent env, read-only, ~1 min):
import glob, json
EJ='/e/data1/datasets/playground/ot-baf/eval_jobs'
EVALS=[('eval-laion_<model>_<safe_dataset>', 300), ...]
for tag, exp in EVALS:
valid=err=0
for f in glob.glob(f'{EJ}/{tag}/*/result.json'):
try: d=json.load(open(f))
except Exception: err+=1; continue
r=((d.get('verifier_result') or {}).get('rewards') or {}).get('reward')
if isinstance(r,(int,float)): valid+=1
else: err+=1
print(f'{tag}: valid={valid} err={err} total={valid+err} / exp {exp}')
Always include the valid/errored/expected matrix in the report. Flag any eval with ERRORED rate ≳10–15% as a re-run candidate.
Check 4 — resuming errored trials (resume_chunked.py)
Resume re-runs only the errored trials of the SAME run dir (keeps valid trials). Standard driver: eval/resume_chunked.py → one unified_eval_listener.py --resume-only --force-reeval per chunk. Restate the same sizing/yaml/conda-env/Pinggy from the original fire or Harbor errors on the config.json conflict.
--force-reeval (NOT --force-eval) — resume-mode flag; distinct from launch-time --force-eval (which bypasses dedup in should_start_job). Bypasses the DB status check AND the active_pairs resume filter (needed when an active job for the same pair uses a different scaffold).--once + --batch-size — in --once mode, freshly-submitted JIDs fold into active_ids as they submit, so the sliding-window --batch-size chain takes effect.
Cat-3 swe-agent (installed-harness) resume — MUST start a Pinggy tunnel. Cat-3 = preferred-harness reproduction (dcagent_eval_config_swe_agent.yaml, swe-agent/openhands/mini-swe-agent) where the sandboxed agent calls back to the served model over a public Pinggy tunnel. The resume branch of eval/jupiter/eval_harbor.sbatch starts the Pinggy SSH tunnel (gated on EVAL_PINGGY_URL) and exports OPENAI_API_BASE/OPENAI_BASE_URL (+ openhands LLM_BASE_URL/LLM_API_KEY, mini-swe-agent MSWEA_*/HOSTED_VLLM_API_BASE). A Cat-3 resume MUST pass the four passthroughs (note hyphens — distinct from launch-mode --pinggy_persistent_url/--pinggy_token):
python eval/resume_chunked.py \
--csv /tmp/resume_cands.csv --preset swebench --org eval \
--tp-size 2 --dp-size 2 --timeout-multiplier 16.0 \
--jobs-dir <EVAL_JOBS_DIR> --conda-env <env> \
--tag-prefix <orig_run_tag_prefix> \
--pinggy-url <free-pair URL, e.g. dadccqeqqf.a.pinggy.link> \
--pinggy-token <free-pair token> \
--config-yaml dcagent_eval_config_swe_agent.yaml \
--agent-parser '' \
--chunk-size 4 --sleep-between 120
--config-yaml = Harbor config (scaffold/parser); --agent-parser '' = disable parser for swe-agent.- Pinggy pair URL+token privileged — read a FREE pair from
.agents/secret.md/pinggy_bank.md.
- terminus-2 resumes leave
EVAL_PINGGY_URL empty → tunnel block is a no-op (they use the normal sbatch path); only Cat-3 installed-harness resumes set the Pinggy flags.
swe-agent retry-policy (dcagent_eval_config_swe_agent.yaml, aligned to M1 SERA-32B 48.67%): ContextLengthExceededError/BadRequestError/AgentEnvironmentTimeoutError/SummarizationTimeout are RETRIED; VerifierOutputParseError/RewardFileEmptyError/RewardFileNotFoundError are terminal. A resume scored under this policy is NOT comparable to one scored under the old drifted config.
Offline-first pre-download: sbatch tries snapshot_download(local_files_only=True) first (HF cache may be read-only when HEAD advanced past the cached snapshot), falls back to online. Pinggy SSH-out is prefixed with proxychains to egress from no-internet nodes. (etag_timeout=120, 600s hard cap on the pre-download wedge.)
Remediations — the §0 audit scopes which to run (re-run §0 after to confirm ✅)
HF trace upload + Supabase registration are normal, sanctioned operations — run the ones the audit flags.
Cross-user FK safety — REQUIRED before EVERY write (update / delete)
The same (model × benchmark) pair can have MULTIPLE sandbox_jobs rows (reruns, resumes, sibling model_ids, AND other users' rows). Before ANY mutation (sandbox_jobs/sandbox_trial_model_usage/models/benchmarks): query ALL rows for the pair, disambiguate by created_at + username, assert the target row is yours (username == me; your usernames: feuer1/bfeuer00/penfever/benjaminfeuer). Scope every update/delete to id+username. NEVER mutate/delete another user's row. When you can't tell which row is yours → STOP and surface, don't guess. (An INSERT of a new own row never touches theirs — see §1.)
1. Manual upload + DB register — manual_db_eval_push.py
Pass the trace_jobs/<RUN_TAG> path (where Harbor writes <task>__<id> trial dirs), NOT eval_jobs/<RUN_TAG> (that only has meta.env). Auto-resolves nested trial subdirs and auto-detects agent/model/benchmark.
set -a; source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"; set +a
cd <OpenThoughts-Agent>
python scripts/database/manual_db_eval_push.py --job-dir trace_jobs/<RUN_TAG> --verbose
- A cross-user-owned (model × benchmark) pair does NOT block registration. When a gate-passing re-fire's pair is already owned by another user, REGISTER A NEW
penfever row alongside it with the corrected score — don't stop, don't ask. The guardrail forbids DELETING/MUTATING a cross-user row; an INSERT never touches theirs (multiple rows per model×benchmark are allowed).
HF trace dataset — use the memory-efficient uploader
Use the streamed, last-episode-per-trial uploader (canonical invocation in rl-agentic-job-cleanup §8) — naive per-conversation extraction loads every episode into RAM (I/O-heavy on GPFS at 300 trials × hundreds of episodes):
python -m scripts.harbor.make_and_upload_trace_dataset \
--job-dir trace_jobs/<RUN_TAG> --repo_id <org>/<RUN_TAG>-traces --episodes last
--episodes last keeps only the scored episode per trial. Eval-vs-RL: RL passes --skip_register; for EVALS register/link the trace repo onto the eval's existing sandbox_jobs row (§2/§3 — set trace/url only, do NOT create a second row or touch the score). On Leonardo, login-node hf upload is SIGKILLed at ~100s → use the sbatch+tunnel pattern (ops/leonardo/ops.md).
2. ⚠️ Verify + fix the MODEL name (with cross-user FK safety)
Auto-detect reads agent_info.model_info.name. For vLLM-served models that field is the vLLM served-model name (numeric, e.g. 1774950145766573), NOT the HF repo → can register a bogus numeric models row. Get the real name:
python3 -c "import json; d=json.load(open('experiments/<RUN_TAG>/configs/<RUN_TAG>_eval_config.json')); print(d['model_hf_name'])"
Then check + repoint (FK-safe; the assert enforces it; models DELETE only if no other-user row FKs it):
c.table("sandbox_jobs").select("model_id,username").eq("id", "<JOB_ID>").execute()
c.table("models").select("name").eq("id", "<MODEL_ID>").execute()
import os; me = os.environ.get("USER")
job = c.table("sandbox_jobs").select("id,username,model_id").eq("id", "<JOB_ID>").execute().data[0]
assert job["username"] == me, f"FK-SAFETY STOP: job owned by {job['username']}, not {me} — do not mutate"
correct = c.table("models").select("id").eq("name", "laion/<real-model-name>").execute()
c.table("sandbox_jobs").update({"model_id": correct.data[0]["id"]}).eq("id", "<JOB_ID>").eq("username", me).execute()
c.table("sandbox_trial_model_usage").update({"model_id": correct.data[0]["id"]}).eq("model_id", "<BOGUS_ID>").execute()
c.table("models").delete().eq("id", "<BOGUS_ID>").execute()
2b. ⚠️ ALSO verify + fix the BENCHMARK name (the version-hash mis-registration)
Same failure mode as the model FK. The harness can register the row under a raw 40-hex dataset version-hash (0c553bd6d05d…/377118ff…/693231ec…) instead of the benchmark name — bites when the launch passed a local hash-named snapshot dir as the dataset. A hash-named benchmark silently MIS-FILES the score and spawns orphan rows. (Fixed for new launches; old legs, the auto-harvest path, and manual registers off such a run dir can still land under a hash.)
Detect (in §0 check-2/3 AND after any register): if the row's benchmark NAME matches ^[0-9a-f]{32,64}$ → mis-keyed.
Resolve from the run-dir name (<benchmark>_<model>_<ts> — the leading segment IS the benchmark) or trace repo (DCAgent2/<benchmark>_…), then map: swebench-verified-random-100-folders=cc1aca76… · dev_set_v2=b94dfab2… · terminal_bench_2=34ab93c4….
Repoint FK-safe:
me = ...
job = c.table("sandbox_jobs").select("id,username,benchmark_id").eq("id","<JOB_ID>").execute().data[0]
assert job["username"] == me, "FK-SAFETY STOP — not your row; surface, do not mutate"
proper = c.table("benchmarks").select("id").eq("name","terminal_bench_2").execute().data[0]["id"]
c.table("sandbox_jobs").update({"benchmark_id": proper}).eq("id","<JOB_ID>").eq("username",me).execute()
Clean the orphan hash benchmark — ONLY if 0 sandbox_jobs reference it across ALL users (cross-user pre-check): delete sandbox_benchmark_tasks CHILDREN first (the FK that blocks the delete; that table has no id column — delete by .eq("benchmark_id", <id>)), then delete the benchmark. LEAVE it if ANY row still references it.
3. Verify it landed
Confirm sandbox_jobs.<JOB_ID>.model_id → real laion/<name> row and trial scores attached. If --skip-hf wasn't used, confirm the trace dataset (DCAgent2/<RUN_TAG>-traces) is non-empty on HF.
4. Clean up disk (only after upload + register verified)
Remove the local eval run dir once traces are on HF + the DB row is correct (the trace_jobs tree is the bulk). Detach a large GPFS rm (nohup/tmux); never du/find to size it first. Do NOT delete shared canonical task dirs.
5. IRIS TPU evals — the idempotent path (GCS-backed, no SLURM)
Iris (marin v6e TPU) evals run through eval/cloud/launch_eval_iris.py, NOT SLURM — so the SLURM mechanics (sacct, cluster run-dirs, sbatch resume, GPFS rm) DON'T apply. The audit logic is identical (same 4 checks + >10% HARD gate + <90%→resume-not-register + cross-user FK safety); only the data source + resume/register MECHANICS differ.
- Enumerate terminal evals (no sacct): the marin jobs table.
iris=/Users/benjaminfeuer/Documents/marin/.venv/bin/iris
$iris --cluster=marin query "SELECT job_id,state FROM jobs WHERE job_id LIKE '%eval-%' ORDER BY job_id DESC" -f csv
State codes: 4=COMPLETED, 5=FAILED, 6=KILLED, 1/2/3=pending/running (SKIP — live). EXEMPT DCAgent2/* measurement runs (§0 pre-gate).
- Results live in GCS, not a filesystem. The aggregate is
<job_output_dir>/<job>/result.json. Resolve the prefix — never guess/scan buckets: OUT=$(python -m hpc.iris.job_output_resolver <job> --cluster …/marin.yaml) (registry-first, iris-fallback). It returns whatever bucket the job wrote to (new single-region gs://marin-us-east5/…, legacy multi-region gs://marin-models-us/…, or the gs://marin-eu-west4 static default). A single gsutil ls "$OUT/<job>/" finds the result.json regardless of region.
- Audit the GCS
result.json — schema is stats.evals.<key> (NOT top-level metrics). Per eval key: score = metrics[0].mean; n_trials/n_total_trials (swe/v2=300, tb2=267); exception_stats[<name>] is a LIST of trial ids → use len() (Σlen over NON-benign names / n_trials = error-fraction; same benign set as §0 check-4). n_cache_tokens=0 = prefix-cache-off fingerprint.
- Resume (<90% OR >10% err) — no sbatch; relaunch
launch_eval_iris.py with the SAME --job_name (Harbor resumes incomplete/errored trials; helpers scripts/iris/check_resume_needed.py + check_progress.py). REGION-CORRECT the relaunch (else it re-lands in eu-west4): run with BOTH export PATH=/Users/benjaminfeuer/Documents/marin/.venv/bin:$PATH (marin iris → region discovery → us-east5) AND the otagent python by FULL PATH /Users/benjaminfeuer/miniconda3/envs/otagent/bin/python (launcher needs omegaconf, which the marin venv lacks). Add --max-retries 2 (a fraction of fresh preemptible v6e-4 slices wedge at model-load model_loader.py:476 ~50min then die). Confirm Region pin: … → us-east5. Iris eval = MAIN Daytona org (), (no snapshot pre-build / cap).
Sibling cleanups: rl-agentic-job-cleanup (RL model), sft-job-cleanup (SFT model), datagen-job-cleanup (trace dataset). Launching evals → eval-agentic-launch (+ the *-iris variant for TPU). Per-cluster particulars → .agents/ops/<cluster>/.
Operating notes
- Stale >12h non-Finished (Started/Running/Pending) rows owned by us are cosmetic cruft → DELETE them (FK-cascade, own-rows-only); registration is create-if-missing so this never blocks a real score landing later.
notes/ot-agent/task_repos/rl_to_check.md is a QUEUE file (flat URL list, one per line, consumed line-by-line by the smoke-test runner; processed entries → rl_checked.md). "Update rl_to_check.md with the fixes" = append new repo URLs, one per line. Do NOT add markdown tables/sections (breaks the parser). Same caution for all task_repos/*.md.
