fine-tuning

name	fine-tuning
description	Complete reference for the fine-tuning pipeline (SFT, KTO, GRPO), cloud HF Jobs workflows, autonomous experiment search, checkpoint evaluation, and LoRA surgery. Covers training CLI flags, YAML configuration, model presets, dataset requirements, LoRA settings, training monitoring, hyperparameter search, and post-training optimization. Use when training models, configuring training runs, choosing hyperparameters, running cloud experiments, inspecting HF jobs, or troubleshooting training issues. This skill is about USING the training system via CLI and YAML — never modifying source code.
allowed-tools	Read, Bash, Write, Grep, Glob

Fine-Tuning Pipeline

Train language models with SFT, KTO, and GRPO locally or on supported cloud providers. This skill also covers the Karpathy-style experiment loop, checkpoint evaluation, LoRA surgery, and the current HF Jobs operational path.

Quick Reference

Task	Command
Interactive menu	./run.sh → Train
Local Docker config run	python tuner.py local-run --job-config Trainers/recipes/<recipe>.yaml --yes
SFT training	cd Trainers/sft && python train_sft.py --model-size 7b
KTO training	cd Trainers/kto && python train_kto.py --model-size 7b
GRPO training	cd Trainers/grpo && python train_grpo.py
Pivot-profile GRPO dataset	cd Trainers/grpo && python train_grpo.py --config configs/pivot_config.yaml --pivot-profile-only
GRPO with pivot filtering	cd Trainers/grpo && python train_grpo.py --config configs/pivot_config.yaml
Env-backed GRPO	cd Trainers/grpo && python train_env_grpo.py --config ./configs/env_config.yaml --dry-run
Experiment loop	python tuner.py experiment-loop --experiment-config configs/flywheel/experiment_loop.yaml
LoRA surgery	python tuner.py surgery --surgery-config configs/lora_surgery.yaml
HF custom job	python tuner.py cloud-run --job-config Trainers/recipes/<recipe>.yaml
Canonical HF train+eval	python tuner.py cloud-pipeline --method sft --preset full
Full experiment bundle	python tuner.py run-experiment --experiment-spec Trainers/cloud/experiments/<spec>.yaml --yes
Evolutionary SFT smoke test	python tuner.py run-experiment --experiment-spec Trainers/cloud/experiments/<evolutionary-spec>.yaml --yes
Staggered experiment batch	python3 scripts/launch_experiment_batch.py Trainers/cloud/experiments/<spec1>.yaml Trainers/cloud/experiments/<spec2>.yaml --yes
One-shot RunPod wrapper job	python3 scripts/runpod_run_job.py --run-tag <tag> --repo-url <git-url> --commit <full-sha> --wrapper <repo-relative.sh> --dry-run
Detached Modal job (survives client exit)	modal run --detach <app_module>::<function>
Blind hardware plan	python tuner.py plan-hardware --experiment-spec Trainers/cloud/experiments/<spec>.yaml
Analyze finished experiment	python tuner.py analyze-experiment --experiment-id latest
Analyze/prune dataset from loss	python3 scripts/prune_dataset_from_loss.py --dataset-path ... --experiment-id ... --analyze-only
Standalone prompt optimization	python tuner.py prompt-optimize --prompt-opt-config configs/prompt_optimization/NAME.yaml
Prompt-optimize SynthChat generation	python -m SynthChat.run generate --prompt-opt-config configs/prompt_optimization/NAME.yaml [options]
Analyze bucket-backed run	python tuner.py bucket analyze --path runs/hf_jobs/sft/<run-prefix>/
Read bucket artifact	python tuner.py bucket read --path runs/.../logs/training_latest.jsonl --jsonl-latest --pretty
List bucket prefix	python tuner.py bucket list --path runs/hf_jobs/sft/<run-prefix>/ --limit 20
Pull bucket prefix locally	python tuner.py bucket pull --path runs/hf_jobs/sft/<run-prefix>/ --dest .
Push local artifact to bucket	python tuner.py bucket push --path local/results.json --dest runs/manual_uploads/
Live HF job list	python tuner.py cloud-jobs list
Live HF job logs	python tuner.py cloud-jobs logs --job professorsynapse/<job-id> --tail 200
Cloud eval against a run	python tuner.py cloud-eval --run latest --preset full
Local deterministic vLLM generation	VLLM_BATCH_INVARIANT=1 python tuner.py batch-generate --engine vllm ...
HF gym against trained model	python tuner.py cloud-gym --run latest --method sft
Warm Space scaffold	python3 Trainers/cloud/scripts/manage_space.py render --template vllm_warm --output-dir /tmp/my-space --base-image ghcr.io/<org>/<image>:<tag>
Warm Space deploy	python3 Trainers/cloud/scripts/manage_space.py deploy --space-id <user>/<space> --template vllm_warm --base-image ghcr.io/<org>/<image>:<tag> --hardware a10g-small --sleep-time 3600 --var BASE_MODEL=<model>
ML training	python tuner.py ml train --config Trainers/ml/configs/templates/regression.yaml

Training Methods at a Glance

Method	Purpose	LR	Epochs	Dataset	When to Use
SFT	Teach format and behavior	2e-4	3	Positive examples only	First stage
KTO	Refine with preferences	1e-6	1	Interleaved True/False	Second stage
GRPO	Optimize against rewards	5e-6	1	Prompts + ground truth	Final online stage
Embedding	Train a retrieval bi-encoder	2e-5	1	Triplets / pairs	Retrieval / RAG embedders

Recommended pipeline: SFT → KTO → GRPO

Embedding training (SentenceTransformer bi-encoders for retrieval) is a separate method with its own registry, dual loader, adapter modes (full/lora/frozen_head), and corpus-level retrieval evaluation. It rides the same local-run/recipe path as SFT. For the full surface use the dedicated embedding-training skill; the triplet/retrieval data shapes are in reference/dataset-formats.md below.

Complexity Tiers

Use --tier on the local SFT and KTO trainers when you want a preset instead of hand-tuning LoRA rank and LR.

Tier	LoRA Rank	LR	Time	Use Case
quick	r=8	5e-4	~5 min	Prototyping and smoke runs
standard	r=64	2e-4	~30-60 min	Normal training
thorough	r=128	1e-4	~2-4 hrs	Quality-focused final runs

Key Directories

• Trainers/sft/ — active SFT trainer
• Trainers/recipes/ — unified training recipe configs (local Docker + HF Jobs); each recipe declares target: local|cloud|both and method: sft|kto|grpo
• Evaluator/recipes/ — unified evaluation recipe configs
• Trainers/kto/ — KTO trainer
• Trainers/grpo/ — GRPO and env-GRPO trainer
• Trainers/embedding/ — embedding (SentenceTransformer bi-encoder) trainer, registry, and dual loader; see the embedding-training skill
• Trainers/archive/legacy_rtx3090/ — archived legacy RTX3090 trainer snapshots and outputs; do not use for new runs
• Datasets/ — JSONL training datasets
• SynthChat/scenarios/ — synthetic data and environment-backed scenarios
• shared/flywheel/ — autonomous experiment-loop logic and LightGBM surrogate search
• Trainers/ml/ — traditional ML / LightGBM training for tabular experiment analysis
• shared/experiment_tracking/ — unified run and experiment tracking

CLI Discipline

• Never cancel a job, delete bucket artifacts, remove files, or relaunch a cost-incurring cloud run unless the user has explicitly approved that exact action in the current conversation.
• Treat cancel/delete/relaunch as irreversible or materially destructive operator actions. Do not infer permission from surrounding context or from a user's broader goal.
• Do not guess command names or flags from memory.
• Before giving command guidance, check tuner/cli/parser.py, tuner/cli/router.py, or the real --help output.
• Prefer repo CLIs and checked-in scripts over ad hoc Python snippets.
• After benchmark runs complete, treat the checked-in benchmark ledger as part of the workflow:
  • model_hardware_benchmark_ledger.md
  • model_hardware_benchmark_ledger.csv
• Treat stage lineage as the source of truth for automation:
  • training: training_lineage.json
  • evaluation: evaluation_lineage.json
  • exact loss: loss_lineage.json
• Treat loss_summary.json as a supporting artifact, not the canonical final loss metadata file.
• The ledger should accumulate real model-size / hardware / timing / cost data so future hardware planning can optimize against observed evidence instead of memory.
• For local trainer iteration, use the checked-in train_sft.py, train_kto.py, and train_grpo.py entrypoints.
• For repeatable local GPU training, prefer python tuner.py local-run --job-config Trainers/recipes/<recipe>.yaml --yes over ad hoc docker run commands. Put the model, dataset, Docker image, package overrides, LoRA settings, training knobs, and artifact paths in YAML.
• For Windows Docker Desktop with GPU, prefer job.transfer: auto or copy in local-run configs. The runner chooses copy mode on Windows because GPU bind mounts can fail with access denied.
• Keep newly released model support in local-run setup.pip pins or image fields. Do not leave one-off package installs in shell history.
• For canonical HF experiments, prefer python tuner.py cloud-pipeline ... over cloud-run.

Research Note Lifecycle

Use .skills/research-reporting/SKILL.md alongside this skill whenever the user wants experiment tracking, research summaries, or reusable post-run analysis.

• When a training run or experiment is launched, create the research note immediately rather than waiting for all stages to finish.
• The initial note should capture config intent from experiment.json and spec_path, with stage_statuses set from the current known state and metrics left null or empty where evidence does not exist yet.
• After training finishes, update the same note with training provenance from training_lineage.json or stage_details.training.
• After evaluation finishes, update the same note with evaluation metrics and observed failure patterns.
• After loss finishes, update the same note with loss metrics, high-loss hashes, and any dataset-review signals.
• After analysis/recommendation finishes, update the same note with hypotheses, selected candidate rank, and recommended next action.
• Do not fork separate notes per stage unless the user explicitly asks for per-stage notes. Default is one note per experiment, updated over time.
• Treat the note as a living artifact for the run: launch state, in-flight updates, and final postmortem should all land in the same document.
• If a stage has not run yet or failed, preserve the section and leave missing values explicit instead of inventing placeholders.

Progressive Reference

Load the specific reference you need:

Reference	When to Load	Path
SFT Training	Running SFT, configuring SFT params	reference/sft-training.md
KTO Training	Running KTO, dataset interleaving, preference tuning	reference/kto-training.md
GRPO Training	Running GRPO, reward config, GSPO variant	reference/grpo-training.md
Model Presets	Choosing models, VRAM planning, LoRA settings	reference/model-presets.md
Dataset Formats	Preparing datasets, format requirements per method	reference/dataset-formats.md
Training Config	YAML config deep-dive	reference/training-config.md
Cloud Training	Provider-native persistence, exact-commit rules, cloud smoke tests	reference/cloud-training.md
Cloud Experiments	Canonical train→eval launches with --train-* overrides	reference/cloud-experiment-launching.md
RunPod Jobs	One-shot wrapper jobs on RunPod pods (non-training lane)	reference/runpod-jobs.md
Modal Jobs	Serverless GPU jobs on Modal; detached runs, crash-proof resume pattern	reference/modal-jobs.md
Checkpoint Evaluation	Best-checkpoint selection via eval	reference/checkpoint-evaluation.md
Experiment Loop	Autonomous hyperparameter search (LLM + LightGBM)	reference/experiment-loop.md
LoRA Techniques	LoRA variants, init methods, config recipes	reference/lora-techniques.md
Evolutionary Config	Experimental gradient-selection config schema and defaults

LoRA Technique Configs

Reference config templates for LoRA variants in .skills/fine-tuning/configs/:

Template	Technique	When to Use
regret_free.yaml	High-rank + rsLoRA + all-linear	Full-FT quality from LoRA
dora.yaml	DoRA (weight-decomposed)	Drop-in quality boost
qlora_dora.yaml	QLoRA + DoRA	Single GPU, VRAM-limited
pissa.yaml	PiSSA (SVD init)	Fast convergence
eva.yaml	EVA (activation SVD init)	Small datasets
olora.yaml	OLoRA (QR init)	Fast convergence, simpler than EVA
loftq.yaml	LoftQ (quant-aware init)	Minimize 4-bit quality loss
grpo_minimal.yaml	Minimal rank for RL	GRPO/reasoning tasks

See reference/lora-techniques.md for full details, integration status, and compatibility notes.

Common Patterns

Quick SFT test run:

cd Trainers/sft
python train_sft.py --model-size 3b --tier quick --dry-run

Config-driven local Docker SFT smoke run:

python tuner.py local-run \
  --job-config Trainers/recipes/qwen35_2b_sft_smoke.yaml \
  --yes

For a different local SFT run, copy a recipe under Trainers/recipes/ (one with target: local or target: both) and change model, dataset, training, lora, job.image, and setup.pip as needed. Use repo-relative local dataset paths; the runner translates them for the container.

Config-driven local Docker embedding smoke run:

python tuner.py local-run \
  --job-config Trainers/recipes/embedding_bge_base_smoke.yaml \
  --yes

Trains a small bge-base-en embedding adapter for a few steps. The recipe rides the modern Unsloth image and layers sentence-transformers + faiss-cpu + datasets via setup.pip (pins are TBD-pending the cloud smoke — captured from the first working run, never invented). For the full embedding surface (registry, adapter modes, retrieval eval) use the embedding-training skill.

Generate a dataset with prompt optimization provenance:

python -m SynthChat.run generate \
  --prompt-opt-config configs/prompt_optimization/synthchat_smoke.yaml \
  --output Datasets/synthchat/prompt_opt_dryrun.jsonl

Use prompt optimization as an opt-in dataset-generation step before training, not as an implicit trainer behavior. SynthChat records the optimizer artifact path and selected candidate ID in row metadata and leaves source YAML unchanged.

Run prompt optimization before dataset generation:

python tuner.py prompt-optimize \
  --prompt-opt-config configs/prompt_optimization/labkit_epistemic_humility_evaluator_smoke.yaml

Prompt optimization is a config-first workflow. Keep subjects, operators, evaluation scenarios, objectives, and stopping rules in configs/prompt_optimization/*.yaml; promote an optimized prompt into canonical scenario/config YAML only after human review. The practical evaluator threshold default is stopping.target_score: 0.8.

The checked-in evaluator smoke config should stay dry-run safe: configs/prompt_optimization/labkit_epistemic_humility_evaluator_smoke.yaml uses evaluation.evaluator.dry_run: true. For a real LM Studio smoke, use an explicit local override or temporary manual copy rather than changing the checked-in config. Keep that real smoke minimal: population_size: 3 and max_generations: 1.

KTO with local dataset:

cd Trainers/kto
python train_kto.py --model-size 7b --local-file ../../Datasets/my_kto_data.jsonl

GRPO continuing from SFT checkpoint:

# Edit configs/config.yaml to set model.lora_path to SFT checkpoint
cd Trainers/grpo
python train_grpo.py

Autonomous hyperparameter search:

python tuner.py experiment-loop \
  --experiment-config configs/flywheel/experiment_loop.yaml \
  --max-experiments 10

Post-training LoRA surgery:

python tuner.py surgery --surgery-config configs/lora_surgery.yaml

Inspect live HF jobs:

python tuner.py cloud-jobs list --limit 20
python tuner.py cloud-jobs show --job professorsynapse/<job-id>
python tuner.py cloud-jobs logs --job professorsynapse/<job-id> --tail 200

Gotcha:

• cloud-jobs show is useful, but for real progress checks prefer the bucket-backed artifacts once they exist. Early raw logs are often just bootstrap noise; the bucket tells you when training/eval/loss has actually started producing useful state.

Read the latest training record from a bucket-backed run:

python tuner.py bucket read \
  --path runs/hf_jobs/sft/<run-prefix>/logs/training_latest.jsonl \
  --jsonl-latest \
  --pretty

Tail the structured progress log or summary artifact directly from the bucket:

python tuner.py bucket analyze \
  --path runs/hf_jobs/sft/<run-prefix>/

python tuner.py bucket read \
  --path runs/hf_jobs/sft/<run-prefix>/logs/training_latest.jsonl \
  --tail 5

python tuner.py bucket read \
  --path runs/hf_jobs/sft/<run-prefix>/logs/stage_summary.json \
  --pretty

python tuner.py bucket list \
  --path runs/hf_jobs/sft/<run-prefix>/ \
  --limit 20

python tuner.py bucket pull \
  --path runs/hf_jobs/sft/<run-prefix>/analysis/loss/ \
  --dest .

python tuner.py bucket push \
  --path local/analysis/notes.json \
  --dest runs/manual_uploads/

Gotcha:

• Use bucket analyze first when a run has finished. It summarizes training_lineage.json, the newest evaluation_lineage.json, and loss_lineage.json in one pass.
• If the same training run has multiple eval reruns or alternate loss prefixes, pass --eval-path and/or --loss-path explicitly so the summary cannot attach to the wrong post-training artifacts.
• If schema pass rate is strong but behavior pass rate is still weak, do not assume you need more tool-call syntax SFT. That pattern usually means the next bottleneck is text-only restraint, clarification, or verify-before-action behavior.

Canonical one-off HF experiment with direct overrides:

python tuner.py cloud-pipeline --method sft --preset full \
  --train-model-name Qwen/Qwen3.5-2B \
  --train-image-profile next \
  --train-gpu a10g-small \
  --train-batch-size 8 \
  --train-gradient-accumulation 4 \
  --train-lora-r 128 \
  --train-lora-alpha 256 \
  --train-use-rslora \
  --train-use-dora \
  --train-no-load-in-4bit \
  --train-lora-target-modules q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
  --yes

python tuner.py cloud-pipeline --method sft \
  --train-model-name Qwen/Qwen3-4B \
  --train-gpu a100-large \
  --train-batch-size 24 \
  --train-gradient-accumulation 2 \
  --train-learning-rate 1e-4 \
  --train-max-steps 60 \
  --train-evolutionary-enabled \
  --train-evolutionary-candidates 4 \
  --train-evolutionary-eval-batch-size 2 \
  --train-evolutionary-validation-config configs/fitness/tool_calling.yaml \
  --train-evolutionary-strategy gradient_noise \
  --train-evolutionary-noise-scale 0.03 \
  --train-evolutionary-max-grad-norm 1.0 \
  --train-evolutionary-selection-method best \
  --train-evolutionary-min-improvement 0.01 \
  --train-evolutionary-eval-frequency 5 \
  --train-evolutionary-warmup-steps 200 \
  --yes

For SFT experiments, the cloud and experiment surfaces now accept:

• --train-save-steps — Override checkpoint save frequency (steps)
• --train-save-total-limit — Override max checkpoints kept
• --train-lora-r
• --train-lora-alpha
• --train-lora-dropout
• --train-use-dora
• --train-use-rslora
• --train-init-lora-weights
• --train-lora-target-modules

Gotcha:

• init_lora_weights=loftq is the only research-style init currently wired for the stable Unsloth SFT path. EVA, PiSSA, and OLoRA still need a separate PEFT-first path.
• lora_target_modules: all-linear is forwarded now, but the legacy Unsloth path is still the riskier runtime. Use an explicit module list for the stable baseline.

Cloud training + eval in one flow:

python tuner.py cloud-pipeline --method sft --preset full

Full experiment with train → eval → exact loss → analysis:

python tuner.py run-experiment \
  --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_full.yaml \
  --yes

Blind hardware planning before launch:

python tuner.py plan-hardware \
  --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_full.yaml \
  --optimize-for balanced

Run experiment with auto hardware selection:

python tuner.py run-experiment \
  --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_full.yaml \
  --auto-hardware \
  --optimize-for cost \
  --yes

Prepare a fixed-tier GPU benchmark matrix for the same model:

python tuner.py plan-hardware --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_benchmark_l4x1.yaml --optimize-for cost
python tuner.py plan-hardware --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_benchmark_a10g_small.yaml --optimize-for cost
python tuner.py plan-hardware --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_benchmark_a100_large.yaml --optimize-for cost

Use this when you want three comparable full-pipeline runs of the same model across increasing GPU tiers. Keep the GPU pinned in the spec, leave training batch/grad accumulation unset, and let run-experiment --auto-hardware fill the batch shape for that tier. Gotcha:

• --auto-hardware can change batch_size and gradient_accumulation per hardware tier. When comparing speed/cost, always record the resolved training shape alongside the GPU flavor. Otherwise you can misread a faster run as “better hardware” when part of the gain came from a larger planner-selected batch.
• run-experiment analysis now appends or updates the checked-in benchmark ledger automatically when analysis runs. If you are comparing ad hoc runs outside experiment orchestration, add or backfill the ledger deliberately.
• When launching several experiment specs in one sweep, do not submit them back-to-back by hand. Use python3 scripts/launch_experiment_batch.py ... --stagger-seconds 5. That avoids same-second job submission races and keeps artifact prefixes easier to track.

Launch a staggered experiment-spec benchmark batch:

python3 scripts/launch_experiment_batch.py \
  Trainers/cloud/experiments/qwen3_4b_full_cycle_benchmark_l40sx1_pruned.yaml \
  Trainers/cloud/experiments/qwen3_4b_full_cycle_benchmark_a100_large_pruned.yaml \
  --auto-hardware \
  --optimize-for cost \
  --yes

Resume or slice the experiment pipeline:

python tuner.py run-experiment --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_full.yaml --from-stage evaluation --yes
python tuner.py run-experiment --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_full.yaml --only-stage loss --yes
python tuner.py run-experiment --experiment-spec Trainers/cloud/experiments/qwen3_4b_full_cycle_full.yaml --skip-stage analysis --skip-stage recommendation --yes

Inspect the finished experiment bundle and next-run candidates:

python tuner.py analyze-experiment --experiment-id latest
python tuner.py analyze-experiment --experiment-id exp_20260321_221651 --json

Analyze a dataset against finished per-example loss before pruning:

python3 scripts/prune_dataset_from_loss.py \
  --dataset-path Datasets/synthchat/my_dataset.jsonl \
  --experiment-id exp_20260322_103122 \
  --analyze-only

Apply a generic pruning rule:

python3 scripts/prune_dataset_from_loss.py \
  --dataset-path Datasets/synthchat/my_dataset.jsonl \
  --experiment-id exp_20260322_103122 \
  --strategy loss_threshold \
  --min-loss 2.0

Apply a capped generic prune budget instead of a hard loss threshold:

python3 scripts/prune_dataset_from_loss.py \
  --dataset-path Datasets/synthchat/my_dataset.jsonl \
  --experiment-id exp_20260322_103122 \
  --strategy top_percent \
  --top-percent 0.02

Apply the current repo-specific result-echo preset and publish in one step:

python3 scripts/prune_dataset_from_loss.py \
  --dataset-path Datasets/synthchat/my_dataset.jsonl \
  --experiment-id exp_20260322_103122 \
  --strategy result_echo_linesdelta_recommendations \
  --publish-repo professorsynapse/claudesidian-synthetic-dataset

Use this only when the analysis report shows that text_only result-echo recap rows are genuinely overrepresented in the high-loss slice. Do not assume that family is always the right prune target.

Environment-backed gym against latest trained adapter on HF:

python tuner.py cloud-gym --run latest --method sft

Tabular LightGBM experiment:

python tuner.py ml train --config Trainers/ml/configs/templates/regression.yaml

Environment Variables

HF_TOKEN=hf_...
WANDB_API_KEY=...
MODAL_TOKEN_ID=...
MODAL_TOKEN_SECRET=...
RUNPOD_API_KEY=...

Output Structure

Local trainers produce timestamped run directories:

{method}_output/YYYYMMDD_HHMMSS/
├── checkpoints/
├── logs/
├── final_model/
└── training_lineage.json

Cloud runs use the canonical provider-native layout:

runs/{provider}/{method}/{timestamp}-{shortsha}/
├── checkpoints/
├── capacity_features.json
├── logs/
├── final_model/
├── training_lineage.json
└── manifest.json

Provider-native storage defaults:

• hf_jobs → Hugging Face Bucket
• modal → Modal Volume
• runpod → RunPod Network Volume

HF Jobs Notes

• Launch from a clean tracked worktree and a pushed commit only; the remote container checks out that exact SHA.
• Keep the main training interpreter compatible with Unsloth and Transformers; any Buckets-only huggingface_hub upgrade must stay isolated from the trainer runtime.
• Keep the bucket-sync overlay self-contained for the Hub client stack: install huggingface_hub>=1.5.0, hf_transfer, and hf_xet into the overlay. Do not rely on the base Unsloth image's hf_xet; older system copies can fail final sync with ImportError: cannot import name 'SKIP_SHA256'.
• Pass HF_TOKEN into the cloud job explicitly; do not assume HF Jobs injects it automatically.
• Treat blank HF_TOKEN / HF_API_KEY values as unset, otherwise bucket sync can fail with Authorization: Bearer .
• For post-training cloud evaluation, prefer python tuner.py cloud-eval --run latest --preset full.
• Keep cloud-eval overlays split by responsibility. The evaluator runtime PYTHONPATH may include lightweight evaluator deps only; Buckets-only packages (huggingface_hub>=1.5.0, hf_transfer, hf_xet) must stay off the evaluator PYTHONPATH and be exposed only through HF_BUCKET_SYNC_PYTHONPATH. Otherwise base-image transformers can import an incompatible Hub client.
• run-experiment now supports stage controls: --only-stage, --from-stage, and repeated --skip-stage.
• run-experiment --auto-hardware uses a blind planner: model size, method, seq length, quantization, and live HF flavor pricing. It does not require prior telemetry.
• plan-hardware is the inspection surface for that same planner.
• When using --auto-hardware, treat the resolved batch_size / gradient_accumulation as part of the experiment definition. Compare wall-clock and cost only after checking those resolved values.
• After training finishes, read training_lineage.json before declaring the hardware/batch shape “good enough.” If peak reserved VRAM is still far below device capacity, the run is underpacked and the next iteration should push batch utilization harder.

Tips

• Always --dry-run first to verify setup without training.
• Use --tier quick for fast prototyping and --tier thorough for final quality runs.
• Use --model-size 3b for fast iteration and 7b for production-style runs.
• SFT with packing: true is much faster.
• KTO datasets must be interleaved True/False.
• GRPO rewards are YAML-driven; edit configs/rewards/, not Python.
• Env-GRPO multi-turn rollouts are token-faithful by default (env_training.token_faithful: true, requires trl>=0.28.0); it emits a per-token env_mask so GRPO trains only on sampled tokens. See reference/grpo-training.md → "Token-Faithful Multi-Turn Rollout".
• fitness.yaml uses FitnessEvaluator for structural validation.
• After training, use surgery only after you have a stable baseline and evaluation scenario.
• training_lineage.json tracks full provenance for reproducibility.