sft-cleanup-hf-only

name

sft-cleanup-hf-only

description

Clean up a completed NON-AGENTIC / HF-only SFT model — HF upload WITHOUT Supabase DB registration. The counterpart to sft-job-cleanup (which uploads AND registers). Use when a completed SFT cell belongs to an HF-only series (config `enable_db_registration: false`, or a known HF-only series like the Delphi #6279 54-grid): upload the weights to laion/ and STOP — do NOT run manual_db_push. Covers cell→hub_model_id resolution (incl. the Delphi launch_54_map.tsv), the Leonardo sbatch-tunnel upload, and the downstream eval hand-off. For DB-registered models use sft-job-cleanup instead.

sft-cleanup-hf-only

Publish the completed HF-only SFT model without Supabase registration. Use sft-job-cleanup for registered SFT.

When to use

• A completed SFT cell whose config sets enable_db_registration: false, OR a known HF-only series (Delphi #6279 delphi-* cells).
• Discovered during a sweep, or "upload the completed Delphi cells (no DB)".
• If it's a normal registered SFT → use sft-job-cleanup (don't skip the DB step there).

Procedure (per completed cell)

1. Confirm HF-only completion. Check enable_db_registration: false (or Delphi delphi-*) and final training progress.
2. Cancel that cell's pending restart chain (squeue … | grep <job> | grep PENDING | awk … | xargs -r scancel).
3. Prepare weights: 8B uses root model.safetensors; 32B/ZeRO-3 needs consolidation. Drop intermediate checkpoint-* and .cache.
4. Resolve hub_model_id:
   • Delphi: name is laion/delphi-<base>-<recipe>_lr1e5-sft; the authoritative cell→hub_model_id↔jobid map is /leonardo_work/AIFAC_5C0_290/bfeuer00/experiments/delphi-prepared-tok/launch_54_map.tsv. Resolve from there (don't guess).
   • Otherwise read the rendered launch config's hub_model_id.
   • Qwen3.5: copy preprocessor_config.json from the base model into the checkpoint before upload. 4b. Check the tokenizer before upload: extra_special_tokens in tokenizer_config.json must be a dict, not a list (python -c "import json;d=json.load(open('<ckpt>/tokenizer_config.json'));assert isinstance(d.get('extra_special_tokens',{}),dict)"). If it is a list, set it to {} and re-save; the RL/SkyRL loader expects a dict.
5. Upload to HF (public default):
   • Leonardo: use the sbatch compute-node + SSH-tunnel upload (.agents/ops/leonardo/ops.md "Leonardo HF Upload"; use hf upload, not upload-large-folder).
   • Jupiter: login node has direct internet → hf upload in tmux.
   • source "$DC_AGENT_SECRET_ENV" for HF_TOKEN (env var only — never inline).
6. Do not run manual_db_push.py.
7. Clean the cell's exp/checkpoint dir after the upload is verified (detached rm on GPFS; no du/find).
8. Verify: the laion/<name> repo exists with model.safetensors (>500MB) + tokenizer/config. Note the cell as done.

Downstream chain (every sweep)

1. SFT completes → HF upload (this skill; no DB).
2. upload completes → eval-standard-launch on the series' eval grid for the newly-uploaded cell(s).
3. eval completes → record scores in the experiment tracker — for Delphi, append the cell's result to /Users/benjaminfeuer/Documents/experiments/active/delphi/rl-scaling-laws-6279/main_sft_evals/SCORES.md. Pull the score from the completed delphi-eval/<RUN>/ output + the eval job's metrics.

Skip work already done.

Delphi eval traps (carry into the eval hand-off)

• HF_HUB_ENABLE_HF_TRANSFER MUST be 0/unset when pre-caching the -sft repos into $HF_HUB_CACHE for eval — hf_transfer isn't in the evalchemy env, so =1 leaves an EMPTY .incomplete blob (silent no-op). Verify the snapshot has model.safetensors (>500MB) before submitting.
• The -sft repos ship the chat template as a separate chat_template.jinja file (NOT in tokenizer_config.json's chat_template key) — the eval sbatch's delphi_v0 override reads/replaces that file.
• Per-cell TP must divide num_attention_heads: 9e18 heads=9→TP=1, 2e19 heads=11→TP=1, 3e19 heads=12→TP=2 (the sbatch's hardcoded TP=2 is WRONG for 9e18/2e19 — pass the optional TP_OVERRIDE 4th positional arg).

Full Delphi series data: project_delphi_sft_hf_only_no_db; upload mechanics: .agents/ops/leonardo/ops.md.