End-to-end LoRA/QLoRA fine-tuning for open-weight models in 2026 — picks a base model from its strengths/weaknesses, assesses whether to train locally or in the cloud and sets up either, runs the training, and visualizes both the dataset and base-vs-tuned outputs. Use when fine-tuning, training a LoRA/QLoRA/DoRA adapter, choosing a base model, deciding local-vs-cloud GPU, or previewing training data and results. NOT for full-parameter pretraining, closed-model API fine-tuning (OpenAI/Gemini), dataset curation from scratch (use fine-tuning-dataset-curator), or RAG/prompt-only work.
End-to-end LoRA/QLoRA fine-tuning for open-weight models in 2026 — picks a base model from its strengths/weaknesses, assesses whether to train locally or in the cloud and sets up either, runs the training, and visualizes both the dataset and base-vs-tuned outputs. Use when fine-tuning, training a LoRA/QLoRA/DoRA adapter, choosing a base model, deciding local-vs-cloud GPU, or previewing training data and results. NOT for full-parameter pretraining, closed-model API fine-tuning (OpenAI/Gemini), dataset curation from scratch (use fine-tuning-dataset-curator), or RAG/prompt-only work.
{"category":"AI & Machine Learning","tags":["lora","qlora","dora","unsloth","axolotl"],"pairs-with":[{"skill":"fine-tuning-dataset-curator","reason":"Curates and cleans the raw data this skill formats and trains on"},{"skill":"ai-engineer","reason":"Deploys and serves the merged adapter produced here"},{"skill":"llm-evaluation-harness","reason":"Rigorous benchmark eval beyond the quick visual diff this skill provides"}],"execution-pattern":"sequential","needs-cdm":true}
io-contract
{"kind":"deliverable","produces":[{"kind":"model-artifact","description":"Trained LoRA/QLoRA adapter, optionally merged and exported to safetensors or GGUF"},{"kind":"report","description":"Hardware assessment, base-model recommendation, and dataset/output visualization HTML reports"},{"kind":"code","description":"Reproducible training config and launch scripts for local or cloud GPUs"}]}
LoRA Training 2026
Fine-tune an open-weight model with a LoRA adapter, made exceptionally easy. This skill
takes you from "I have some data and a goal" to a trained, previewed, exportable adapter —
choosing the base model, the hardware (your machine or a rented GPU), and showing you what
your data and your results actually look like at every step.
When to Use
✅ Use for:
Fine-tuning an open-weight LLM (Llama, Qwen, Gemma, Mistral, Phi, DeepSeek, SmolLM…) with LoRA/QLoRA/DoRA
Choosing which base model fits your task, license, language, and VRAM budget
Deciding whether to train locally or in the cloud, then setting up whichever fits
Previewing/sanity-checking a training dataset before spending GPU hours
Comparing base-model vs fine-tuned outputs side by side after a run
❌ NOT for:
Full-parameter pretraining or continued pretraining from scratch (different scaling regime)
Fine-tuning closed models via API (OpenAI, Gemini, Claude) — no LoRA, no weights
Building a dataset from raw sources — delegate to fine-tuning-dataset-curator
Prompt engineering or RAG when no weight update is actually needed
Rigorous benchmark evaluation — delegate to llm-evaluation-harness
Core Process
flowchart TD
A[Goal + raw data] --> B{Is LoRA the right tool?}
B -->|Need new knowledge at scale| Z[Reconsider: RAG or pretraining]
B -->|Need new behavior/format/style| C[1. Select base model]
C --> D[2. Assess compute: local vs cloud]
D -->|Fits locally| E[Setup local env]
D -->|Too big / no GPU| F[Setup cloud GPU]
E --> G[3. Prepare + visualize dataset]
F --> G
G --> H{Data looks healthy?}
H -->|No| G
H -->|Yes| I[4. Train LoRA/QLoRA]
I --> J[5. Visualize base vs tuned outputs]
J --> K{Good enough?}
K -->|Overfit / underfit| L[Tune hyperparams] --> I
K -->|Yes| M[6. Merge + export safetensors/GGUF]
Step 1 — Select the base model
Match the task to a model's strengths, license, size, and context length — never default to
"whatever is trending." Run the selector or read the registry:
This ranks candidates from references/base-models-2026.md. For deep trade-offs (instruct vs base,
MoE vs dense, vision, reasoning distillation), read that reference. When in doubt, delegate the
nuanced call to the agent.
base-model-selector
Step 2 — Assess compute (local vs cloud)
Run hardware detection; it returns a verdict and a ready-to-run setup path for either route:
It estimates VRAM for the chosen model+method, reads your GPU, and prints LOCAL OK or
USE CLOUD with a concrete provider recommendation and launch command. The compute-advisor
agent handles ambiguous cases (e.g. "I have a 12 GB card but want a 32B model"). See
references/local-vs-cloud.md for the VRAM math, provider table, and cost comparison.
Step 3 — Prepare and visualize the dataset
Convert to the trainer format, split, and look at it before training:
The visualizer reports token-length distributions, role balance, duplicates, length outliers,
and renders sample conversations. Open the HTML; do not skip this — most failed runs are bad-data
runs. The dataset-doctor agent diagnoses anything the report flags.
Step 4 — Train
One trainer drives local and cloud the same way (Unsloth/PEFT under the hood):
The training-orchestrator agent writes run.yaml, picks rank/alpha/lr/scheduler from the
dataset size and method, and watches the loss. See references/hyperparameters.md.
Step 5 — Visualize results
Generate a side-by-side base-vs-tuned preview on held-out prompts:
The eval-visualizer agent reads this to judge overfit/underfit/regressions and recommends the
next move (more data, fewer epochs, lower lr, merge-and-ship).
Exports a merged adapter for Ollama/llama.cpp (GGUF) or vLLM/TGI (safetensors).
Anti-Patterns
Anti-Pattern: "Bigger base model = better adapter"
Novice: "My task is hard, so I'll LoRA a 70B model."
Expert: Adapter quality is dominated by data quality and task–model fit, not raw parameter
count. A clean 1–2k-example set on Qwen3-8B usually beats a noisy set on Llama-4-70B, trains in
minutes on one consumer GPU, and serves cheaply. Scale the base only when the base can't do the
task even with perfect prompting. Start small, prove the loop, then scale.
Timeline: Pre-2024 the instinct was "scale solves it." Since QLoRA (2023) and the strong 7–9B
instruct models of 2025–2026, the bottleneck moved decisively to data.
Anti-Pattern: Training before looking at the data
Novice: "The JSONL parses, so it's fine — start the run."
Expert: Parsing ≠ healthy. Run visualize_dataset.py first: look for label leakage, truncated
examples past your seq-len, role imbalance, near-duplicates inflating apparent size, and a length
tail that silently triples your token bill. Most "the model got dumber" outcomes are a 30-second
data look that nobody took.
Timeline: Always true; the 2026 tooling just makes the look fast enough that skipping it is
inexcusable.
Anti-Pattern: Reaching for cloud GPUs by reflex
Novice: "Fine-tuning needs a datacenter, so rent an 8×H100 node."
Expert: With 4-bit QLoRA, an 8B model trains comfortably on a single 16 GB consumer GPU, and a
12–14B on 24 GB. assess_hardware.py tells you when local genuinely works — which is most small-to-mid
LoRA jobs. Cloud is for >24B, multi-GPU, or no-GPU laptops, not a default. Conversely, don't try to
full-fine-tune a 32B locally and conclude "LoRA doesn't work."
Timeline: 2026 consumer cards (16–24 GB) plus Unsloth's memory kernels made local the default
for adapters; cloud-by-reflex is a 2022 habit.
Anti-Pattern: rank/alpha cargo-culting
Novice: "The blog used rank 8, alpha 16, so I will too."
Expert: Rank sets adapter capacity; for style/format tasks r=8–16 is plenty, for harder behavior
shifts r=32–64. Set alpha ≈ 2×rank (or use rsLoRA so scaling is rank-stable), target attention
and MLP projections, and tune learning rate to the method (2e-4 QLoRA is a starting point, lower
for DoRA). Copying a number without the regime behind it under- or over-fits silently.
Timeline: rsLoRA/DoRA (2024) changed the alpha intuition; the old "alpha=16 always" advice is stale.
Bundle Map
Agents (agents/) — load the one matching the step
Agent
Use when
base-model-selector.md
Picking among open-weight models by task/license/VRAM/language trade-offs
compute-advisor.md
Deciding local vs cloud and producing the exact setup for the chosen route
dataset-doctor.md
A dataset visualization flagged problems and you need them diagnosed + fixed
training-orchestrator.md
Authoring the run config, choosing hyperparameters, and supervising the run
eval-visualizer.md
Interpreting the base-vs-tuned comparison and deciding the next move
References (references/) — NOT loaded by default
File
Consult when
base-models-2026.md
Comparing source models and their strengths/weaknesses
