accelerate

# HuggingFace Accelerate - Unified Distributed Training ## Quick start Accelerate simplifies distributed training to 4 lines of code. **Installation**: ```bash pip install accelerate ``` **Convert PyTorch script** (4 lines): ```python import torch + from accelerate import Accelerator + accelerator = Accelerator() model = torch.nn.Transformer() optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset) + model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: optimizer.zero_grad() loss = model(batch) - loss.backward() + accelerator.backward(loss) optimizer.step() ``` **Run** (single command): ```bash accelerate launch train.py ``` ## Common workflows ### Workflow 1: From single GPU to multi-GPU **Original script**: ```python # train.py import torch model = torch.nn.Linear(10, 2).to('cuda') optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) for epoch in range(10): for batch in dataloader: batch = batch.to('cuda') optimizer.zero_grad() loss = model(batch).mean() loss.backward() optimizer.step() ``` **With Accelerate** (4 lines added): ```python # train.py import torch from accelerate import Accelerator # +1 accelerator = Accelerator() # +2 model = torch.nn.Linear(10, 2) optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # +3 for epoch in range(10): for batch in dataloader: # No .to('cuda') needed - automatic! optimizer.zero_grad() loss = model(batch).mean() accelerator.backward(loss) # +4 optimizer.step() ``` **Configure** (interactive): ```bash accelerate config ``` **Questions**: - Which machine? (single/multi GPU/TPU/CPU) - How many machines? (1) - Mixed precision? (no/fp16/bf16/fp8) - DeepSpeed? (no/yes) **Launch** (works on any setup): ```bash # Single GPU accelerate launch train.py # Multi-GPU (8 GPUs) accelerate launch --multi_gpu --num_processes 8 train.py # Multi-node accelerate launch --multi_gpu --num_processes 16 \ --num_machines 2 --machine_rank 0 \ --main_process_ip $MASTER_ADDR \ train.py ``` ### Workflow 2: Mixed precision training **Enable FP16/BF16**: ```python from accelerate import Accelerator # FP16 (with gradient scaling) accelerator = Accelerator(mixed_precision='fp16') # BF16 (no scaling, more stable) accelerator = Accelerator(mixed_precision='bf16') # FP8 (H100+) accelerator = Accelerator(mixed_precision='fp8') model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # Everything else is automatic! for batch in dataloader: with accelerator.autocast(): # Optional, done automatically loss = model(batch) accelerator.backward(loss) ``` ### Workflow 3: DeepSpeed ZeRO integration **Enable DeepSpeed ZeRO-2** (pass a `DeepSpeedPlugin`, not a raw dict): ```python from accelerate import Accelerator, DeepSpeedPlugin deepspeed_plugin = DeepSpeedPlugin( zero_stage=2, # ZeRO-2 offload_optimizer_device="none", # or "cpu" to offload gradient_accumulation_steps=4, ) accelerator = Accelerator( mixed_precision='bf16', deepspeed_plugin=deepspeed_plugin, # DeepSpeedPlugin instance (or dict[str, DeepSpeedPlugin]) ) # Same code as before! model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) ``` **Or point at a full DeepSpeed JSON config via the plugin**: ```python from accelerate import Accelerator, DeepSpeedPlugin # hf_ds_config accepts a path to a DeepSpeed config JSON (or a dict) deepspeed_plugin = DeepSpeedPlugin(hf_ds_config="ds_config.json") accelerator = Accelerator(mixed_precision='bf16', deepspeed_plugin=deepspeed_plugin) ``` **ds_config.json** (a raw DeepSpeed config — passed via the plugin, NOT via `--config_file`): ```json { "fp16": {"enabled": false}, "bf16": {"enabled": true}, "zero_optimization": { "stage": 2, "offload_optimizer": {"device": "cpu"}, "allgather_bucket_size": 5e8, "reduce_bucket_size": 5e8 } } ``` **Or via interactive config**: ```bash accelerate config # Select: DeepSpeed → ZeRO-2 # This writes an accelerate YAML config (default: ~/.cache/huggingface/accelerate/default_config.yaml) ``` **Launch** (`--config_file` expects an accelerate YAML, not a raw DeepSpeed JSON): ```bash # Uses the default accelerate config written by `accelerate config` accelerate launch train.py # Or point at a specific accelerate YAML accelerate launch --config_file accelerate_deepspeed.yaml train.py ``` ### Workflow 4: FSDP (Fully Sharded Data Parallel) **Enable FSDP**: ```python from accelerate import Accelerator, FullyShardedDataParallelPlugin fsdp_plugin = FullyShardedDataParallelPlugin( sharding_strategy="FULL_SHARD", # ZeRO-3 equivalent auto_wrap_policy="transformer_based_wrap", # valid: transformer_based_wrap | size_based_wrap | no_wrap cpu_offload=False ) accelerator = Accelerator( mixed_precision='bf16', fsdp_plugin=fsdp_plugin ) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) ``` **Or via config**: ```bash accelerate config # Select: FSDP → Full Shard → No CPU Offload ``` ### Workflow 5: Gradient accumulation **Accumulate gradients**: ```python from accelerate import Accelerator accelerator = Accelerator(gradient_accumulation_steps=4) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: with accelerator.accumulate(model): # Handles accumulation optimizer.zero_grad() loss = model(batch) accelerator.backward(loss) optimizer.step() ``` **Effective batch size**: `batch_size * num_gpus * gradient_accumulation_steps` ## When to use vs alternatives **Use Accelerate when**: - Want simplest distributed training - Need single script for any hardware - Use HuggingFace ecosystem - Want flexibility (DDP/DeepSpeed/FSDP/Megatron) - Need quick prototyping **Key advantages**: - **4 lines**: Minimal code changes - **Unified API**: Same code for DDP, DeepSpeed, FSDP, Megatron - **Automatic**: Device placement, mixed precision, sharding - **Interactive config**: No manual launcher setup - **Single launch**: Works everywhere **Use alternatives instead**: - **PyTorch Lightning**: Need callbacks, high-level abstractions - **Ray Train**: Multi-node orchestration, hyperparameter tuning - **DeepSpeed**: Direct API control, advanced features - **Raw DDP**: Maximum control, minimal abstraction ## Common issues **Issue: Wrong device placement** Don't manually move to device: ```python # WRONG batch = batch.to('cuda') # CORRECT # Accelerate handles it automatically after prepare() ``` **Issue: Gradient accumulation not working** Use context manager: ```python # CORRECT with accelerator.accumulate(model): optimizer.zero_grad() accelerator.backward(loss) optimizer.step() ``` **Issue: Checkpointing in distributed** Use accelerator methods: ```python # Save only on main process if accelerator.is_main_process: accelerator.save_state('checkpoint/') # Load on all processes accelerator.load_state('checkpoint/') ``` **Issue: Different results with FSDP** Ensure same random seed: ```python from accelerate.utils import set_seed set_seed(42) ``` ## Advanced topics **Megatron integration**: See [references/megatron-integration.md](references/megatron-integration.md) for tensor parallelism, pipeline parallelism, and sequence parallelism setup. **Custom plugins**: See [references/custom-plugins.md](references/custom-plugins.md) for creating custom distributed plugins and advanced configuration. **Performance tuning**: See [references/performance.md](references/performance.md) for profiling, memory optimization, and best practices. ## Hardware requirements - **CPU**: Works (slow) - **Single GPU**: Works - **Multi-GPU**: DDP (default), DeepSpeed, or FSDP - **Multi-node**: DDP, DeepSpeed, FSDP, Megatron - **TPU**: Supported - **Apple MPS**: Supported **Launcher requirements**: - **DDP**: `torch.distributed.run` (built-in) - **DeepSpeed**: `deepspeed` (pip install deepspeed) - **FSDP**: PyTorch 1.12+ (built-in) - **Megatron**: Custom setup ## Resources - Docs: https://huggingface.co/docs/accelerate - GitHub: https://github.com/huggingface/accelerate - Version: 1.11.0+ - Tutorial: "Accelerate your scripts" - Examples: https://github.com/huggingface/accelerate/tree/main/examples - Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries