unsloth-fine-tuning

# Unsloth - Fast LLM Fine-Tuning Fine-tune LLMs 2-5x faster with 50-80% less VRAM. Supports SFT, RL (GRPO), vision, TTS, and 300+ models with zero accuracy loss. ## When to Use Unsloth **Use Unsloth when:** - Fine-tuning on a single GPU with LoRA/QLoRA (consumer or datacenter) - Training reasoning models with GRPO, Dr. GRPO, DAPO, BNPO, or GSPO - Fine-tuning vision models (Qwen3-VL, Gemma 3, Llama 3.2 Vision) - Fine-tuning TTS models (Orpheus, Sesame-CSM, Whisper) - Exporting to GGUF for Ollama, llama.cpp, or LM Studio - Need padding-free training and uncontaminated packing (automatic) - Using FP8 precision for additional memory savings on Ampere+ GPUs **Don't use Unsloth when:** - Multi-node distributed training at scale (Unsloth DDP works but is single-node) - Apple Silicon / MLX (not yet supported) - Full fine-tuning of 70B+ models (use DeepSpeed + Transformers) - Custom architectures not supported by `transformers` - Cloud-managed training without GPU access (use Tinker instead) **Unsloth vs Alternatives:** | Need | Use | |------|-----| | Fast single-GPU LoRA/QLoRA | **Unsloth** | | Managed cloud LoRA training | Tinker | | Parameter-efficient methods (IA3, Prefix, etc.) | PEFT | | Multi-node distributed training | DeepSpeed + Transformers | | YAML-config-driven training | Axolotl | | Full fine-tuning with FSDP | Transformers + Accelerate | ## Quick Reference | Topic | Documentation | |-------|---------------| | Overview & Features | [docs/overview.md](docs/overview.md) | | Installation (pip) | [docs/installation-pip.md](docs/installation-pip.md) | | Installation (Docker) | [docs/installation-docker.md](docs/installation-docker.md) | | Model Selection Guide | [docs/model-selection.md](docs/model-selection.md) | | VRAM Requirements | [docs/requirements.md](docs/requirements.md) | | Model Catalog (300+) | [docs/models.md](docs/models.md) | | Datasets & Formatting | [docs/datasets.md](docs/datasets.md) | | Chat Templates | [docs/chat-templates.md](docs/chat-templates.md) | | LoRA Hyperparameters | [docs/lora-hyperparameters.md](docs/lora-hyperparameters.md) | | GRPO RL Tutorial | [docs/tutorial-grpo.md](docs/tutorial-grpo.md) | | Advanced RL Parameters | [docs/advanced-rl.md](docs/advanced-rl.md) | | Memory-Efficient RL | [docs/memory-efficient-rl.md](docs/memory-efficient-rl.md) | | Vision Fine-Tuning | [docs/vision-fine-tuning.md](docs/vision-fine-tuning.md) | | Vision RL (VLM GRPO) | [docs/vision-rl.md](docs/vision-rl.md) | | TTS Fine-Tuning | [docs/tts-fine-tuning.md](docs/tts-fine-tuning.md) | | Saving to GGUF | [docs/saving-to-gguf.md](docs/saving-to-gguf.md) | | Saving to Ollama | [docs/saving-to-ollama.md](docs/saving-to-ollama.md) | | vLLM Deployment | [docs/vllm-guide.md](docs/vllm-guide.md) | | FP8 Training | [docs/fp8-rl.md](docs/fp8-rl.md) | | FP16 vs BF16 for RL | [docs/fp16-vs-bf16.md](docs/fp16-vs-bf16.md) | | Multi-GPU DDP | [docs/multi-gpu-ddp.md](docs/multi-gpu-ddp.md) | | Kernels & Packing | [docs/kernels-packing.md](docs/kernels-packing.md) | | Inference | [docs/inference.md](docs/inference.md) | | Troubleshooting | [docs/troubleshooting-faq.md](docs/troubleshooting-faq.md) | | Troubleshooting Inference | [docs/troubleshooting-inference.md](docs/troubleshooting-inference.md) | ## Installation ```bash # Recommended (pip) pip install unsloth # With vLLM (for GRPO fast inference) pip install uv && uv pip install unsloth vllm # Docker (all dependencies pre-installed) docker run -d -e JUPYTER_PASSWORD="mypassword" \ -p 8888:8888 --gpus all -v $(pwd)/work:/workspace/work \ unsloth/unsloth ``` **Requirements:** Linux or Windows (WSL), NVIDIA GPU with CUDA Capability 7.0+ (V100, T4, RTX 20-50, A100, H100, L40). AMD and Intel GPUs also supported. Python 3.10-3.13. --- ## Workflow 1: SFT (Supervised Fine-Tuning) Use this for standard instruction tuning, chat fine-tuning, or domain adaptation. ### Checklist - [ ] Prepare dataset in ShareGPT, ChatML, or Alpaca format - [ ] Choose base vs instruct model (see Model Selection below) - [ ] Select QLoRA (4-bit) or LoRA (16-bit) based on VRAM - [ ] Set hyperparameters (rank, alpha, LR, epochs) - [ ] Run training with SFTTrainer - [ ] Save and deploy (LoRA adapter, merged 16-bit, or GGUF) ### Implementation ```python from unsloth import FastLanguageModel from trl import SFTTrainer, SFTConfig from datasets import load_dataset # Step 1: Load model (QLoRA 4-bit) model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Qwen3-8B-bnb-4bit", # or any HF model max_seq_length=2048, load_in_4bit=True, # False for LoRA 16-bit ) # Step 2: Add LoRA adapters model = FastLanguageModel.get_peft_model( model, r=16, # Rank: 8-128 (16-32 recommended) lora_alpha=16, # Alpha: equal to r or 2*r lora_dropout=0, # 0 is default, use 0.05-0.1 for regularization target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], use_gradient_checkpointing="unsloth", # 30% less VRAM use_rslora=False, # True for rank-stabilized LoRA ) # Step 3: Prepare dataset dataset = load_dataset("philschmid/dolly-15k-oai-style", split="train") # Step 4: Train trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=dataset, args=SFTConfig( output_dir="./sft-output", per_device_train_batch_size=2, gradient_accumulation_steps=4, # Effective batch = 2*4 = 8 num_train_epochs=3, learning_rate=2e-4, fp16=True, # or bf16=True logging_steps=10, optim="adamw_8bit", max_seq_length=2048, packing=True, # Uncontaminated packing (2-5x faster) ), ) trainer.train() # Step 5: Save model.save_pretrained("lora_adapter") # LoRA only (~6MB) tokenizer.save_pretrained("lora_adapter") ``` ### Data Formats | Format | Template | Use Case | |--------|----------|----------| | ShareGPT | `{"conversations": [{"from": "human", ...}]}` | Multi-turn chat, instruct models | | ChatML / OpenAI | `{"messages": [{"role": "user", ...}]}` | OpenAI-compatible, instruct models | | Alpaca | `{"instruction": ..., "input": ..., "output": ...}` | Single-turn tasks, base models | | Raw text | Plain text corpus | Continued pretraining | Use `get_chat_template(tokenizer, chat_template="chatml")` to apply templates. Use `standardize_sharegpt(dataset)` for ShareGPT-formatted data with non-standard keys. ### Training on Completions Only Mask user inputs so loss is only computed on assistant responses: ```python from unsloth.chat_templates import train_on_responses_only trainer = train_on_responses_only( trainer, instruction_part="<|start_header_id|>user<|end_header_id|>\n\n", # Llama 3.x response_part="<|start_header_id|>assistant<|end_header_id|>\n\n", ) # For Gemma: instruction_part="<start_of_turn>user\n", response_part="<start_of_turn>model\n" ``` **Sources:** [docs/datasets.md](docs/datasets.md), [docs/chat-templates.md](docs/chat-templates.md), [docs/lora-hyperparameters.md](docs/lora-hyperparameters.md) --- ## Workflow 2: RL Training (GRPO) Use this for training reasoning models with reward functions — math, code, format compliance, verifiable tasks. ### Checklist - [ ] Define reward function(s) returning float scores - [ ] Choose model and enable vLLM fast inference - [ ] Enable Unsloth Standby for memory-efficient RL - [ ] Configure GRPOConfig with num_generations, epsilon, loss_type - [ ] Monitor reward curves and KL divergence - [ ] Save and export model ### Implementation ```python import os os.environ["UNSLOTH_VLLM_STANDBY"] = "1" # Memory-efficient RL from unsloth import FastLanguageModel import torch import re model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Qwen3-8B", max_seq_length=2048, load_in_4bit=True, # False for LoRA 16-bit fast_inference=True, # Enable vLLM for fast generation max_lora_rank=32, gpu_memory_utilization=0.9, # Reduce if OOM ) model = FastLanguageModel.get_peft_model( model, r=32, lora_alpha=64, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], use_gradient_checkpointing="unsloth", ) # Define reward functions def correctness_reward(completions, answer, **kwargs): scores = [] for completion in completions: match = re.search(r"<answer>(.*?)</answer>", completion, re.DOTALL) extracted = match.group(1).strip() if match else "" scores.append(1.0 if extracted == answer else 0.0) return scores def format_reward(completions, **kwargs): pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>" return [1.0 if re.search(pattern, c, re.DOTALL) else 0.0 for c in completions] # Train from trl import GRPOConfig, GRPOTrainer training_args = GRPOConfig( output_dir="./grpo-output", per_device_train_batch_size=2, gradient_accumulation_steps=4, learning_rate=5e-6, num_generations=8, # Rollouts per prompt max_completion_length=512, max_prompt_length=512, max_steps=250, temperature=1.0, # RL algorithm variants loss_type="dapo", # "grpo", "dr_grpo", "dapo", "bnpo" epsilon=0.2, epsilon_high=0.28, # DAPO upper clipping scale_rewards="none", # Dr. GRPO: no reward scaling optim="adamw_8bit", report_to="none", ) trainer = GRPOTrainer( model=model, processing_class=tokenizer, args=training_args, train_dataset=dataset, reward_funcs=[correctness_reward, format_reward], ) trainer.train() # Save model.save_lora("grpo_saved_lora") ``` ### RL Algorithm Variants | Algorithm | `loss_type` | Key Setting | Notes | |-----------|-------------|-------------|-------| | GRPO | `"grpo"` | Default | Standard group relative policy optimization | | Dr. GRPO | `"dr_grpo"` | `scale_rewards="none"` | No reward normalization, more stable | | DAPO | `"dapo"` | `epsilon_high=0.28` | Two-sided clipping, recommended default | | BNPO | `"bnpo"` | — | Bounded negative policy optimization | | GSPO | any | `importance_sampling_level="sequence"` | Sequence-level importance weighting (Qwen team) | ### Unsloth Standby (Memory-Efficient RL) Set `os.environ["UNSLOTH_VLLM_STANDBY"] = "1"` before imports. This shares vLLM's weight space with training and repurposes KV cache memory during training — saving up to 60% VRAM. On H100 80GB: 16GB shared weights + 64GB multi-purpose space. **Sources:** [docs/tutorial-grpo.md](docs/tutorial-grpo.md), [docs/advanced-rl.md](docs/advanced-rl.md), [docs/memory-efficient-rl.md](docs/memory-efficient-rl.md) --- ## Workflow 3: Vision Fine-Tuning Use this for training vision-language models on image+text tasks. ### Implementation ```python from unsloth import FastVisionModel from trl import SFTTrainer, SFTConfig from unsloth.trainer import UnslothVisionDataCollator model, tokenizer = FastVisionModel.from_pretrained(