crud-archive-run

name

crud-archive-run

description

Durably ARCHIVE everything informative from a finished run / experiment before it's cleaned up or its cluster artifacts age out — ALL Harbor trace_jobs (raw per-trial traces), ALL ray logs, ALL stdout/stderr (incl. vLLM/serving logs), and wandb. Pack-rat by design: if it's potentially informative, keep it. Only skip the non-informative-or-huge (model weights/checkpoints, core/memory dumps, massive raw tmux-pane / terminal-recording bytes). Many-tiny-files → tar THEN rsync (never rsync thousands of small files raw). Use when concluding/archiving an experiment, before a cleanup skill `rm`s an on-disk tree, or before CoreWeave R2/pod artifacts get GC'd. Per-run-type component maps live below; WHERE each artifact lives per cluster is a pointer into `.agents/ops/<cluster>/` and `.agents/projects/{harbor,marinskyrl,ot-agent}/`.

crud-archive-run

Archive potentially informative artifacts; skip only files that are both non-informative and large.

✅ ARCHIVE (always — every run type)

• All raw Harbor traces — every per-trial dir under trace_jobs/ (RL) / eval_jobs/<name>/ (eval) / the datagen trace dir: result.json, config.json, manifest.json, lock.json, agent/trajectory.json (the raw agent transcript — INFORMATIVE, keep even at multi-MB), verifier/ (reward.txt, ctrf.json, test-stdout.txt), step_results, trajectory.summarization-*.json.
• All ray logs — the ray session dir (raylet, gcs_server, per-worker *.out/*.err, python-core-*).
• All stdout / stderr — SLURM .out/.err, the complete CoreWeave finelog, vllm.log, job.log, and per-trial trial.log.
• wandb — the local wandb/ run dir if present; else record the run URL/id in the archive's MANIFEST.
• Configs / launch command / rendered YAML / metric CSVs / trainer_log.jsonl.

❌ SKIP (non-informative AND large)

• Model weights / checkpoints — *.safetensors, *.pt, *.bin, global_step_*/, consolidated shards (they live on HF / R2; not useful for post-hoc debugging).
• Core / memory dumps — core.*, *.hprof, coredump trees.
• Massive raw terminal-pane bytes — *.pane (raw tmux pane dumps) and agent/recording.cast (asciinema) only when large and redundant with trajectory.json; keep small casts.
• Conda/uv/pip caches, extracted wheel trees, __pycache__, .venv.

Mechanic — tar many-small-files, then rsync

On the source cluster/pod, tar the small-file tree before rsyncing it:

# on the cluster (SLURM) — one tarball per run, excluding the SKIP set
tar --exclude='*.safetensors' --exclude='*.pt' --exclude='*.bin' --exclude='global_step_*' \
    --exclude='core.*' --exclude='*.pane' \
    -czf /tmp/<run>_archive.tgz -C <run_dir> trace_jobs logs *.log config* wandb  # adjust to what exists
rsync -aP <cluster>:/tmp/<run>_archive.tgz  <dest>/         # then rm the /tmp tarball

Keep large single logs (vllm.log) in the tarball or rsync them alongside. Verify the tarball is non-empty and lists the expected trees (tar tzf … | head) before deleting the source.

Per-run-type components — WHAT + WHERE (pointers, they drift — read the ops/projects doc)

• CoreWeave agentic RL (SkyRL/MarinSkyRL) — durable traces: s3://marin-us-east-02a/iris/<job>/trace_jobs (--trials-dir auto; pull with aws s3 --endpoint-url <R2>); pod-local traces: /app/experiments/<run>/trace_jobs (grab before pod GC with scripts/iris/analyze_coreweave_rl_job_live.sh <pod> cp). Full log: iris … job logs --since-ms <submit> --no-tail. Ray logs and vllm.log are pod-local; record the W&B URL.
• SFT (LLaMA-Factory / axolotl, SLURM) — .out per-step logs at experiments/<job>/logs/*.out, trainer_log.jsonl, rendered config, wandb. Weights → HF (SKIP). Log path via scontrol show job <id> -o StdOut=/%Z. Details: .agents/projects/{llama-factory,axolotl}/, the cluster ops doc.
• Datagen (Harbor traces) — one-level trace_jobs/<trial>/result.json + the harbor run log; the artifact is the trace set (→ HF), but archive the trace_jobs + logs. Details: .agents/projects/harbor/.
• Eval (agentic Harbor) — eval_jobs/<name>/<trial>/{result.json,config.json,agent/trajectory.json, verifier/,manifest.json,lock.json} + top-level vllm.log, job.log, per-trial trial.log. Skip the big recording.cast/*.pane when large. Details: .agents/projects/harbor/, eval-agentic-cleanup.
• Cluster paths: .agents/ops/iris/ (CoreWeave), .agents/ops/tacc/ (SLURM), and .agents/ops/empireai/ (SLURM). Read the relevant one first.

Destination

Default: ~/Documents/experiments/<active|complete>/<exp>/run_archive/<run-id>/. Write a one-line MANIFEST (run-id, cluster, job-id, dates, kept/skipped artifacts, W&B URL). Optionally push the tarball to HF (penfever/…-archive, public default laion/) or R2. Archive and verify before any cleanup reclaims the tree.