unsloth-quantization

name	unsloth-quantization
description	Utilizing Dynamic 4-bit quantization, FP8 training, and 8-bit optimizers to minimize VRAM usage without sacrificing accuracy. Triggers: quantization, dynamic 4-bit, fp8, bitsandbytes, adamw_8bit, qat.

Overview

Unsloth utilizes advanced quantization techniques to reduce the memory footprint of LLM fine-tuning. This includes "Dynamic 4-bit" loading (protecting sensitive layers), FP8 training for modern GPUs, and the use of 8-bit optimizers to save gigabytes of VRAM.

When to Use

• When training on GPUs with limited VRAM (e.g., 8GB, 12GB, or 16GB).
• When aiming for the fastest possible training speeds on H100 or RTX 40 series GPUs.
• When trying to balance model size and reasoning performance.

Decision Tree

1. Is your GPU RTX 40 series or newer (Ada/Hopper)?
   • Yes: Use FP8 Dynamic for 2x faster training.
   • No: Use BF16/FP16.
2. Running out of VRAM?
   • Yes: Ensure load_in_4bit=True and use adamw_8bit optimizer.
3. Is accuracy dropping significantly?
   • Yes: Use "Dynamic" variants that protect the first and last layers.

Workflows

FP8 Training Configuration

1. Select a model variant ending in '-FP8-Dynamic'.
2. Configure the trainer to use the FP8 backend (available for H100 and Ada Lovelace architectures).
3. Verify speedup, which typically reaches 2x compared to standard BF16.

VRAM-Constrained Training Setup

1. Set load_in_4bit=True and use_gradient_checkpointing='unsloth'.
2. Apply 'adamw_8bit' optimizer in TrainingArguments.
3. Set per_device_train_batch_size to 1 and maximize .