Menü
Run PyTorch training across GPUs with minimal changes.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Query and edit a SiYuan knowledge base via its API.
Create, read, edit Excel .xlsx spreadsheets and CSVs.
Create, read, edit Excel .xlsx spreadsheets and CSVs.
Curate LLM training data: dedupe, filter, PII redaction.
Scrape sites with stealth browsing and Cloudflare bypass.
Clean training loops with built-in distributed support.
Basierend auf der SOC-Berufsklassifikation
| name | huggingface-accelerate |
| description | Run PyTorch training across GPUs with minimal changes. |
| version | 1.0.0 |
| author | Orchestra Research |
| license | MIT |
| dependencies | ["accelerate","torch","transformers"] |
| platforms | ["linux","macos","windows"] |
| metadata | {"hermes":{"tags":["Distributed Training","HuggingFace","Accelerate","DeepSpeed","FSDP","Mixed Precision","PyTorch","DDP","Unified API","Simple"]}} |
Accelerate 将分布式训练简化为仅需 4 行代码即可实现。
安装方式：
pip install accelerate
转换 PyTorch 脚本（4 行）：
import torch
+ from accelerate import Accelerator
+ accelerator = Accelerator()
model = torch.nn.Transformer()
optimizer = torch.optim.Adam(model.parameters())
dataloader = torch.utils.data.DataLoader(dataset)
+ model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
for batch in dataloader:
optimizer.zero_grad()
loss = model(batch)
- loss.backward()
+ accelerator.backward(loss)
optimizer.step()
运行（单条命令）：
accelerate launch train.py
原始脚本：
# train.py
import torch
model = torch.nn.Linear(10, 2).to('cuda')
optimizer = torch.optim.Adam(model.parameters())
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
for epoch in range(10):
for batch in dataloader:
batch = batch.to('cuda')
optimizer.zero_grad()
loss = model(batch).mean()
loss.backward()
optimizer.step()
启用 Accelerate 功能后（新增 4 行）：
# train.py
import torch
from accelerate import Accelerator # +1
accelerator = Accelerator() # +2
model = torch.nn.Linear(10, 2)
optimizer = torch.optim.Adam(model.parameters())
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # +3
for epoch in range(10):
for batch in dataloader:
# No .to('cuda') needed - automatic!
optimizer.zero_grad()
loss = model(batch).mean()
accelerator.backward(loss) # +4
optimizer.step()
配置（交互模式）：
accelerate config
问题：
启动（适用于任何配置环境）：
# Single GPU
accelerate launch train.py
# Multi-GPU (8 GPUs)
accelerate launch --multi_gpu --num_processes 8 train.py
# Multi-node
accelerate launch --multi_gpu --num_processes 16 \
--num_machines 2 --machine_rank 0 \
--main_process_ip $MASTER_ADDR \
train.py
启用 FP16/BF16：
from accelerate import Accelerator
# FP16 (with gradient scaling)
accelerator = Accelerator(mixed_precision='fp16')
# BF16 (no scaling, more stable)
accelerator = Accelerator(mixed_precision='bf16')
# FP8 (H100+)
accelerator = Accelerator(mixed_precision='fp8')
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
# Everything else is automatic!
for batch in dataloader:
with accelerator.autocast(): # Optional, done automatically
loss = model(batch)
accelerator.backward(loss)
启用 DeepSpeed ZeRO-2：
from accelerate import Accelerator
accelerator = Accelerator(
mixed_precision='bf16',
deepspeed_plugin={
"zero_stage": 2, # ZeRO-2
"offload_optimizer": False,
"gradient_accumulation_steps": 4
}
)
# Same code as before!
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
或通过配置文件：
accelerate config
# Select: DeepSpeed → ZeRO-2
deepspeed_config.json：
深度速度配置文件。
{
"fp16": {"enabled": false},
"bf16": {"enabled": true},
"zero_optimization": {
"stage": 2,
"offload_optimizer": {"device": "cpu"},
"allgather_bucket_size": 5e8,
"reduce_bucket_size": 5e8
}
}
启动：
accelerate launch --config_file deepspeed_config.json train.py
启用 FSDP：
from accelerate import Accelerator, FullyShardedDataParallelPlugin
fsdp_plugin = FullyShardedDataParallelPlugin(
sharding_strategy="FULL_SHARD", # ZeRO-3 equivalent
auto_wrap_policy="TRANSFORMER_AUTO_WRAP",
cpu_offload=False
)
accelerator = Accelerator(
mixed_precision='bf16',
fsdp_plugin=fsdp_plugin
)
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
或通过配置文件：
accelerate config
# Select: FSDP → Full Shard → No CPU Offload
累积梯度：
from accelerate import Accelerator
accelerator = Accelerator(gradient_accumulation_steps=4)
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
for batch in dataloader:
with accelerator.accumulate(model): # Handles accumulation
optimizer.zero_grad()
loss = model(batch)
accelerator.backward(loss)
optimizer.step()
有效批量大小：batch_size * num_gpus * gradient_accumulation_steps
适合使用 Accelerate 的情况：
主要优势：
应选择替代方案的情况：
问题：设备分配错误
请勿手动将模型移动到目标设备：
# WRONG
batch = batch.to('cuda')
# CORRECT
# Accelerate handles it automatically after prepare()
问题：梯度累积功能无法正常使用
请使用上下文管理器：
# CORRECT
with accelerator.accumulate(model):
optimizer.zero_grad()
accelerator.backward(loss)
optimizer.step()
问题：分布式环境下的检查点功能
请使用加速器方法：
# Save only on main process
if accelerator.is_main_process:
accelerator.save_state('checkpoint/')
# Load on all processes
accelerator.load_state('checkpoint/')
问题：使用 FSDP 时结果不一致
请确保随机种子相同：
from accelerate.utils import set_seed
set_seed(42)
Megatron集成：如需了解张量并行、流水线并行及序列并行配置方法，请参阅references/megatron-integration.md。
自定义插件：如需创建自定义分布式插件并进行高级配置，请参阅references/custom-plugins.md。
性能调优：有关性能分析、内存优化及最佳实践的内容，请参阅references/performance.md。
启动器要求：
torch.distributed.run（内置功能）deepspeed（需通过pip安装deepspeed）