datagen-reduce-dataset-snapshots

name

datagen-reduce-dataset-snapshots

description

Reduce the Daytona snapshot (unique-environment) count of a Harbor task dataset below the cap by editing its patcher's environment-build logic, without breaking task quality. Use when a dataset is flagged "SnapshotCapExceeded" / "N unique environments" with N over the threshold (target < 10), e.g. swegym at 906. The loop: set snapshot+oracle thresholds → count → diagnose the env-hash driver → group/unionize Dockerfiles in the patcher → regenerate + upload → re-count → TWO-TIER quality gate (harbor infra smoke + `--stages oracle` gold-patch yield) → navigate the snapshot↔fidelity tradeoff within a bounded iteration budget → record. Runs LOCALLY on the Mac + Daytona (no GPU).

datagen-reduce-dataset-snapshots

Harbor's Daytona backend builds one container snapshot per unique environment directory, keyed by a content hash of environment/ — which for our patchers is just environment/Dockerfile (solution/tests/metadata live in sibling dirs and don't affect the hash). Daytona enforces a HARD org cap (40) and a per-launch max_new_snapshots (10). A dataset whose tasks each render a distinct Dockerfile explodes to ~1 snapshot/task and is unlaunchable. Fix: make the patcher render a small shared set of Dockerfiles (grouped by a coarse key like Python version); clone the repo + run repo-specific install at agent/verifier runtime instead of image-build time, so thousands of tasks collapse onto a handful of environments.

Thresholds — set BEFORE regenerating (step 0, write in the log)

Snapshot reduction is lossy: fewer envs → less each task's env is tailored → some repos' installs stop reproducing the gold patch → oracle yield drops. You are choosing an operating point on the snapshot↔fidelity curve, not "fixing a bug" — decide these up front to avoid an unbounded chase:

• Snapshot ceiling: hard < 10 (ideally ≤ 8). Non-negotiable — it's the cap. (20 is the "skip the dataset" line; this skill pulls a dataset back under it.)
• Oracle-yield floor: a number set up front (e.g. ≥ 80% for a dataset destined for RL/datagen verification; lower only with explicit reason). Without a pre-set floor every result looks "one more round will help" and you over-fit get_specs to the 40-task sample.
• Iteration budget: max regenerate→oracle rounds (e.g. 2–3). Each round is a full regenerate + upload + oracle sample (slow + Daytona builds); diminishing returns set in fast once the easy repo-family fixes are in.
• Sample size for the yield estimate: 40 is a usable read; don't re-sample endlessly chasing a ±few-point wobble — that is not signal.

Authoritative count tool

scripts/harbor/count_snapshots_from_tasks.py computes the exact content-hash dedup count Daytona's auto_snapshot path uses (get_task_environment_hash / analyze_task_dockerfiles). Run it on a local task dir (post-extraction or post-generation), not a live HF id, to skip the registry round-trip:

PY=/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python
$PY -m scripts.harbor.count_snapshots_from_tasks --local-dataset <tasks_dir>
# read the "UNIQUE ENVIRONMENTS (SNAPSHOTS): N" line

For an uploaded HF dataset, extract first:

$PY -m scripts.datagen.extract_tasks_from_parquet \
  --parquet <hf-id> --output_dir /tmp/snapcount-<slug> --on_exist overwrite
$PY -m scripts.harbor.count_snapshots_from_tasks --local-dataset /tmp/snapcount-<slug>

The cycle

1. Count the current artifact. Extract the flagged HF dataset → count. Confirm it's genuinely over threshold — use the tool above, not row counts.

2. Diagnose the env-hash driver. Find the patcher. Most live in the shared data/patchers/ dir (patch_<name>_tasks.py, patch_exp_rpt_*_tasks.py, patch_mix_h*_tasks.py, patch_code_contests_tasks.py, …); only a few datasets keep a per-source-subdir patcher (data/<name>/generate_patched.py, e.g. swegym, swesmith). ls data/patchers/ | grep -i <name>; ls -d data/<name> 2>/dev/null. The unique-env count == number of distinct rendered Dockerfile strings. The explosion almost always comes from per-task interpolation into the Dockerfile: repo@commit in a build-time git clone/RUN, a per-instance base image, or per-task apt pins. Generate a small sample with the current patcher and count it — if the uploaded artifact is high but a fresh sample is low, the artifact is just stale (made by an older patcher) and step 4 is a pure regenerate+reupload.

3. Rewrite the env logic to group (only if step 2's fresh sample is still high). Make the Dockerfile depend on a coarse grouping key (e.g. Python version), not the task. The swegym pattern:

   • Base image + a union of system apt deps per group; nothing task-specific in the Dockerfile.
   • Move git clone <repo>@<commit> and repo-specific pip install/make into instruction.md (agent setup), solution/solve.sh, tests/test.sh — they run at trial time, against the shared image.
   • Keep a get_specs(repo, version) map so each repo still gets its correct Python
     • install command; only the Dockerfile-affecting part is coarsened. Re-generate a sample → count → iterate the grouping until < 10.

4. Regenerate the full dataset + upload to a NEW repo. Never overwrite the validated artifact; bump the version suffix (...-validated-v2 → ...-v3, or ...-snap-reduced). Run the patcher with (no limit), (public per ). Then re-extract + re-count the repo to confirm < 10 end-to-end.

Decision at each round (don't iterate reflexively)

1. Oracle ≥ floor at < 10 snapshots → SHIP (record both numbers; done).
2. Oracle < floor, failures cluster on a few repo-families with identifiable missing install steps → one targeted get_specs round (add the apt pkg / pip constraint / install command for those families), regenerate, re-oracle. Spend a budget slot.
3. Oracle < floor, failures spread evenly (systemic — the shared env can't satisfy the repo diversity) → the inherent tradeoff, not a bug. Do NOT keep tweaking get_specs. Pick a coarser-but-larger grouping that still fits the cap (e.g. group by py-version × repo-family → maybe 8–15 envs instead of 5) and re-measure the (snapshots, oracle) point. Present the tradeoff curve.
4. Budget exhausted and still < floor at any ≤-cap grouping → STOP and surface to the user with the curve (e.g. "5 envs → 48% oracle; 12 → 74%; 30 → 91% but busts the 10-cap"). Shipping a lower-fidelity dataset, raising the floor, or shelving is the user's call, not an infinite loop's.

Worked example — swegym (#31, 906 → 5 snapshots, a FAILED reduction)

laion/swegym-tasks-patched-validated-v2: 989 tasks → 906 unique envs (≈1:1). The patcher data/swegym/generate_patched.py already groups Dockerfiles by Python version (get_specs() interpolates only {python_version} + {extra_packages} from a fixed apt_map, clones the repo at runtime), so v2 was a stale artifact from an older per-task patcher. Regenerating with the current patcher → laion/swegym-tasks-patched-validated-v3 → 906 → 5 snapshots (no patcher code change). But the quality gate exposed the tradeoff: Tier-1 harbor smoke = 100% infra (looks great, would be mistaken for success); Tier-2 oracle = 19/40 = 47.5% — 5 shared py-version envs can't satisfy every repo's install. v3 is snapshots-green but oracle-red = a FAILED reduction, not shippable.

Guardrails

• Never ship below the oracle floor just because snapshots are green. A tiny-snapshot dataset whose gold patches don't verify is worse than useless for RL/datagen (the reward signal is broken). Snapshots-green + oracle-red is a failed reduction, recorded as such.
• Never raise/bypass the Daytona cap (max_new_snapshots, max_org_snapshots) or convert SnapshotCapExceeded to a warning — reduce the real count (feedback_daytona_snapshot_caps_hard_limit).
• Never overwrite the existing validated artifact — always a new versioned repo.
• Uploads are PUBLIC by default to laion/; enable_db_registration stays off (these are task datasets, not models).
• On the Mac, run everything with the otagent python (/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python); source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}" first (.agents/secret.md).