lora-training-2026

license	Apache-2.0
name	lora-training-2026
description	End-to-end LoRA/QLoRA fine-tuning for open-weight models in 2026 — picks a base model from its strengths/weaknesses, assesses whether to train locally or in the cloud and sets up either, runs the training, and visualizes both the dataset and base-vs-tuned outputs. Use when fine-tuning, training a LoRA/QLoRA/DoRA adapter, choosing a base model, deciding local-vs-cloud GPU, or previewing training data and results. NOT for full-parameter pretraining, closed-model API fine-tuning (OpenAI/Gemini), dataset curation from scratch (use fine-tuning-dataset-curator), or RAG/prompt-only work.
allowed-tools	Read,Write,Edit,Bash(python:*,pip:*,uv:*,nvidia-smi:*,git:*,huggingface-cli:*,modal:*),Grep,Glob
argument-hint	[task: select-model|assess-compute|prepare-data|train|visualize] [dataset-or-model]
category	AI & Machine Learning
tags	["lora","qlora","fine-tuning","peft","local-vs-cloud","visualization"]
metadata	{"category":"AI & Machine Learning","tags":["lora","qlora","dora","unsloth","axolotl"],"pairs-with":[{"skill":"fine-tuning-dataset-curator","reason":"Curates and cleans the raw data this skill formats and trains on"},{"skill":"ai-engineer","reason":"Deploys and serves the merged adapter produced here"},{"skill":"llm-evaluation-harness","reason":"Rigorous benchmark eval beyond the quick visual diff this skill provides"}],"execution-pattern":"sequential","needs-cdm":true}
io-contract	{"kind":"deliverable","produces":[{"kind":"model-artifact","description":"Trained LoRA/QLoRA adapter, optionally merged and exported to safetensors or GGUF"},{"kind":"report","description":"Hardware assessment, base-model recommendation, and dataset/output visualization HTML reports"},{"kind":"code","description":"Reproducible training config and launch scripts for local or cloud GPUs"}]}

LoRA Training 2026

Fine-tune an open-weight model with a LoRA adapter, made exceptionally easy. This skill takes you from "I have some data and a goal" to a trained, previewed, exportable adapter — choosing the base model, the hardware (your machine or a rented GPU), and showing you what your data and your results actually look like at every step.

When to Use

✅ Use for:

• Fine-tuning an open-weight LLM (Llama, Qwen, Gemma, Mistral, Phi, DeepSeek, SmolLM…) with LoRA/QLoRA/DoRA
• Choosing which base model fits your task, license, language, and VRAM budget
• Deciding whether to train locally or in the cloud, then setting up whichever fits
• Previewing/sanity-checking a training dataset before spending GPU hours
• Comparing base-model vs fine-tuned outputs side by side after a run

❌ NOT for:

• Full-parameter pretraining or continued pretraining from scratch (different scaling regime)
• Fine-tuning closed models via API (OpenAI, Gemini, Claude) — no LoRA, no weights
• Building a dataset from raw sources — delegate to fine-tuning-dataset-curator
• Prompt engineering or RAG when no weight update is actually needed
• Rigorous benchmark evaluation — delegate to llm-evaluation-harness

---

Core Process

flowchart TD
  A[Goal + raw data] --> B{Is LoRA the right tool?}
  B -->|Need new knowledge at scale| Z[Reconsider: RAG or pretraining]
  B -->|Need new behavior/format/style| C[1. Select base model]
  C --> D[2. Assess compute: local vs cloud]
  D -->|Fits locally| E[Setup local env]
  D -->|Too big / no GPU| F[Setup cloud GPU]
  E --> G[3. Prepare + visualize dataset]
  F --> G
  G --> H{Data looks healthy?}
  H -->|No| G
  H -->|Yes| I[4. Train LoRA/QLoRA]
  I --> J[5. Visualize base vs tuned outputs]
  J --> K{Good enough?}
  K -->|Overfit / underfit| L[Tune hyperparams] --> I
  K -->|Yes| M[6. Merge + export safetensors/GGUF]

Step 1 — Select the base model

Match the task to a model's strengths, license, size, and context length — never default to "whatever is trending." Run the selector or read the registry:

python scripts/recommend_base_model.py --task "support-bot reply rewriting" \
  --vram 16 --license-need permissive --languages en,es

This ranks candidates from references/base-models-2026.md. For deep trade-offs (instruct vs base, MoE vs dense, vision, reasoning distillation), read that reference. When in doubt, delegate the nuanced call to the base-model-selector agent.

Step 2 — Assess compute (local vs cloud)

Run hardware detection; it returns a verdict and a ready-to-run setup path for either route:

python scripts/assess_hardware.py --model qwen3-8b --method qlora --seq-len 4096

It estimates VRAM for the chosen model+method, reads your GPU, and prints LOCAL OK or USE CLOUD with a concrete provider recommendation and launch command. The compute-advisor agent handles ambiguous cases (e.g. "I have a 12 GB card but want a 32B model"). See references/local-vs-cloud.md for the VRAM math, provider table, and cost comparison.

Step 3 — Prepare and visualize the dataset

Convert to the trainer format, split, and look at it before training:

python scripts/prepare_dataset.py raw.jsonl --format chatml --split 0.9 --out data/
python scripts/visualize_dataset.py data/train.jsonl --out reports/dataset.html

The visualizer reports token-length distributions, role balance, duplicates, length outliers, and renders sample conversations. Open the HTML; do not skip this — most failed runs are bad-data runs. The dataset-doctor agent diagnoses anything the report flags.

Step 4 — Train

One trainer drives local and cloud the same way (Unsloth/PEFT under the hood):

python scripts/train_lora.py --config configs/run.yaml          # local
bash   scripts/train_lora.sh  --provider modal --config configs/run.yaml   # cloud

The training-orchestrator agent writes run.yaml, picks rank/alpha/lr/scheduler from the dataset size and method, and watches the loss. See references/hyperparameters.md.

Step 5 — Visualize results

Generate a side-by-side base-vs-tuned preview on held-out prompts:

python scripts/compare_outputs.py --base <model> --adapter out/adapter \
  --prompts data/eval.jsonl --out reports/compare.html

The eval-visualizer agent reads this to judge overfit/underfit/regressions and recommends the next move (more data, fewer epochs, lower lr, merge-and-ship).

Step 6 — Merge and export

python scripts/merge_and_export.py --base <model> --adapter out/adapter \
  --format gguf --quant q4_k_m --out exports/

Exports a merged adapter for Ollama/llama.cpp (GGUF) or vLLM/TGI (safetensors).

---

Anti-Patterns

Anti-Pattern: "Bigger base model = better adapter"

Novice: "My task is hard, so I'll LoRA a 70B model." Expert: Adapter quality is dominated by data quality and task–model fit, not raw parameter count. A clean 1–2k-example set on Qwen3-8B usually beats a noisy set on Llama-4-70B, trains in minutes on one consumer GPU, and serves cheaply. Scale the base only when the base can't do the task even with perfect prompting. Start small, prove the loop, then scale. Timeline: Pre-2024 the instinct was "scale solves it." Since QLoRA (2023) and the strong 7–9B instruct models of 2025–2026, the bottleneck moved decisively to data.

Anti-Pattern: Training before looking at the data

Novice: "The JSONL parses, so it's fine — start the run." Expert: Parsing ≠ healthy. Run visualize_dataset.py first: look for label leakage, truncated examples past your seq-len, role imbalance, near-duplicates inflating apparent size, and a length tail that silently triples your token bill. Most "the model got dumber" outcomes are a 30-second data look that nobody took. Timeline: Always true; the 2026 tooling just makes the look fast enough that skipping it is inexcusable.

Anti-Pattern: Reaching for cloud GPUs by reflex

Novice: "Fine-tuning needs a datacenter, so rent an 8×H100 node." Expert: With 4-bit QLoRA, an 8B model trains comfortably on a single 16 GB consumer GPU, and a 12–14B on 24 GB. assess_hardware.py tells you when local genuinely works — which is most small-to-mid LoRA jobs. Cloud is for >24B, multi-GPU, or no-GPU laptops, not a default. Conversely, don't try to full-fine-tune a 32B locally and conclude "LoRA doesn't work." Timeline: 2026 consumer cards (16–24 GB) plus Unsloth's memory kernels made local the default for adapters; cloud-by-reflex is a 2022 habit.

Anti-Pattern: rank/alpha cargo-culting

Novice: "The blog used rank 8, alpha 16, so I will too." Expert: Rank sets adapter capacity; for style/format tasks r=8–16 is plenty, for harder behavior shifts r=32–64. Set alpha ≈ 2×rank (or use rsLoRA so scaling is rank-stable), target attention and MLP projections, and tune learning rate to the method (2e-4 QLoRA is a starting point, lower for DoRA). Copying a number without the regime behind it under- or over-fits silently. Timeline: rsLoRA/DoRA (2024) changed the alpha intuition; the old "alpha=16 always" advice is stale.

---

Bundle Map

Agents (agents/) — load the one matching the step

Agent	Use when
base-model-selector.md	Picking among open-weight models by task/license/VRAM/language trade-offs
compute-advisor.md	Deciding local vs cloud and producing the exact setup for the chosen route
dataset-doctor.md	A dataset visualization flagged problems and you need them diagnosed + fixed
training-orchestrator.md	Authoring the run config, choosing hyperparameters, and supervising the run
eval-visualizer.md	Interpreting the base-vs-tuned comparison and deciding the next move

References (references/) — NOT loaded by default

File	Consult when
base-models-2026.md	Comparing source models and their strengths/weaknesses
local-vs-cloud.md	VRAM math, provider table, cost comparison, setup recipes
hyperparameters.md	Choosing rank/alpha/lr/scheduler/target-modules; QLoRA/DoRA/rsLoRA
dataset-formats.md	Formatting chatml/instruction/DPO/completion/vision data
visualization-guide.md	Reading the dataset and comparison HTML reports
troubleshooting.md	OOM, loss spikes/NaNs, overfitting, garbage outputs, slow training

Scripts (scripts/) — all self-contained, --help on each

Script	Does
recommend_base_model.py	Ranks base models from the registry for your task/constraints
assess_hardware.py	Detects GPU/VRAM, estimates need, prints LOCAL OK or USE CLOUD + setup
prepare_dataset.py	Validates, converts, dedups, and splits a dataset into trainer format
visualize_dataset.py	HTML report: token lengths, role balance, dupes, sample render
train_lora.py	Unified LoRA/QLoRA/DoRA trainer (Unsloth→PEFT fallback) driven by YAML
train_lora.sh	One-shot cloud launcher (Modal / RunPod / generic SSH) wrapping the trainer
compare_outputs.py	Side-by-side base-vs-adapter generations → HTML diff report
merge_and_export.py	Merges the adapter and exports safetensors or GGUF (quantized)
model_registry.json	Data backing recommend_base_model.py (editable source of truth)

Assets (assets/)

File	Consult when
run.example.yaml	Starting a new run config; copy and edit rather than writing one from scratch

---

Quality Checklist (before declaring done)

• Base model chosen with an explicit reason (task fit + license + VRAM), not by trend
• assess_hardware.py run; local-vs-cloud decision is recorded, not assumed
• visualize_dataset.py report opened and reviewed; no truncation/leak/dup flags
• Eval prompts are held out of training data
• compare_outputs.py shows improvement on the target behavior without regressing general ability
• Run config (run.yaml) committed for reproducibility
• Exported in the format the serving target actually consumes (GGUF for Ollama, safetensors for vLLM)