fine-tune-hyperparameter-search

name

fine-tune-hyperparameter-search

description

Design, run, analyze, and package machine-learning fine-tuning experiments with disciplined hyperparameter search. Use when the user wants to fine-tune an ML model, tune learning rate/batch size/optimizer/regularization/LoRA or adapter parameters, configure Optuna/Ray/W&B/Hugging Face Trainer/Keras/scikit-learn sweeps, compare trials, avoid overfitting, select a final checkpoint, or produce a reproducible training recipe and search report.

Fine-Tune Hyperparameter Search

Use this skill to turn model fine-tuning into a controlled experimental loop. The goal is not to run many trials blindly; the goal is to spend compute where it reduces uncertainty and produces a reproducible final model.

Core Workflow

1. Frame the tuning problem

   • Identify task, model family, dataset, metric, deployment constraints, hardware, time/compute budget, and acceptable risk.
   • Ask at most one blocking question if target metric, dataset split, or compute budget is unclear.
   • Decide whether the goal is best validation score, robustness, latency/size tradeoff, cost reduction, or stable reproducibility.

2. Audit the training setup

   • Verify train/validation/test separation, label quality, leakage risks, class imbalance, sequence/image sizes, preprocessing, and baseline.
   • Establish a fixed evaluation protocol before tuning.
   • Do not tune on the test set.

3. Choose tuning mode

   • Use a manual pilot when the training stack is unknown or fragile.
   • Use random or low-discrepancy search for broad early exploration.
   • Use Bayesian/Optuna-style search when trials are moderately expensive and metrics are smooth enough.
   • Use successive halving/ASHA/Hyperband when many bad trials can be stopped early.
   • Read references/search-spaces.md for parameter ranges and references/tuning-protocol.md for search strategy.

4. Build the search space

   • Start with high-impact parameters: learning rate, batch size/effective batch size, epochs/steps, weight decay, warmup, scheduler, dropout, augmentation, optimizer, seed, and architecture/adaptation knobs.
   • For LLM/adapter fine-tuning, include LoRA rank/alpha/dropout, target modules, sequence length, packing, gradient accumulation, precision, and clipping.
   • Use log scales for learning rate, weight decay, dropout-adjacent regularization, and batch-size-like capacity when appropriate.

5. Run trials as experiments

   • Give each trial a stable ID, config, seed, code version, data version, hardware note, metric output, and failure reason if any.
   • Save checkpoints only when useful; avoid filling disk with losing trials.
   • Record both optimization metric and safety/quality guardrails.

6. Analyze and prune

   • Compare against the baseline and trivial heuristics.
   • Look for variance across seeds before trusting a narrow win.
   • Prune configs that only win by overfitting, leakage, too much compute, or unacceptable latency/memory.

7. Finalize

   • Retrain or resume the best config according to the chosen protocol.
   • Evaluate once on the held-out test set, if available.
   • Export the final config, checkpoint path, metrics, limitations, and exact reproduction command.

Output Modes

• Search plan: objective, metric, budget, search space, scheduler, and stop rules.
• Config generation: produce YAML/JSON sweep configs for the user's stack.
• Trial analysis: rank runs, diagnose sensitivity, and recommend next trials.
• Training-code integration: patch existing training scripts to expose tunable parameters and logging.
• Final recipe: best config, retraining command, evaluation command, and reproducibility note.

Required Experiment Ledger

For non-trivial tuning, maintain:

• Dataset/version
• Split policy
• Base model/checkpoint
• Metric to optimize
• Guardrail metrics
• Budget
• Search algorithm
• Search space
• Trial log
• Best config
• Final evaluation
• Known risks

Helper Script

Use scripts/make_sweep_plan.py to generate a starter JSON search plan:

python skills/ml-training/fine-tune-hyperparameter-search/scripts/make_sweep_plan.py \
  --task llm-lora \
  --budget small \
  --metric eval_loss \
  --direction minimize

Treat the output as a starting point. Adapt it to the user's framework and constraints before running expensive trials.

Guardrails

• Do not tune against the test set.
• Do not compare runs with different data splits unless that is the experiment.
• Do not report a single lucky seed as robust improvement.
• Do not ignore failed or pruned trials; failures are evidence about the feasible region.
• Do not expand the search space after seeing test results.
• Do not optimize one metric while silently degrading required guardrail metrics.
• Do not recommend expensive sweeps before running a cheap baseline and sanity check.