メニュー
Guide for debugging distributed training issues in AReaL. Use when user encounters hangs, wrong results, OOM, or communication errors.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/areal-project/AReaL --skill debug-distributedコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Read-only pull request review workflow with risk analysis, targeted checklists, and Codex subagent consultation.
Upgrade focused runtime dependencies in AReaL. First validates and updates per-package API checklists for structural completeness, then updates pyproject files, resolves conflicts, locks, updates the Dockerfile, and audits API compatibility against the checklists.
AReaL commit message conventions. MUST load on every git commit -- provides Conventional Commits format with scope inference from file paths.
SKILL.md を表示中
SOC 職業分類に基づく
| name | debug-distributed |
| description | Guide for debugging distributed training issues in AReaL. Use when user encounters hangs, wrong results, OOM, or communication errors. |
Debugging guide for distributed training issues in AReaL (FSDP2, TP, CP, EP).
This skill is triggered when:
Always follow the minimal demo principle: Reproduce with the least amount of code to narrow down the issue faster.
# Bad: Debug in full training loop
# Good: Create minimal script
import torch
import torch.distributed as dist
dist.init_process_group("nccl")
rank = dist.get_rank()
# Reproduce the exact operation that fails
tensor = torch.ones(10).cuda()
dist.all_reduce(tensor) # <-- Isolate the failing op
print(f"Rank {rank}: {tensor}")
Reduction strategy:
Environment Variables for Debugging:
# Full debug logging
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
# torch.compile debugging
export TORCH_LOGS="+dynamo,recompiles"
export TORCHDYNAMO_VERBOSE=1
Dump Call Stack with py-spy (for hung processes):
# Find process IDs
ps aux | grep python
# Dump call stack of specific rank
py-spy dump --pid <PID>
# Record flame graph for performance analysis
py-spy record -o profile.svg --pid <PID> --duration 30
Common Causes:
all_reduce, another doesn't.Debug Steps:
# Verify group membership
mesh = parallel_dims.get_mesh("dp_shard_cp")
group = mesh.get_group()
print(f"Rank {dist.get_rank()}: group size = {dist.get_world_size(group)}")
# Print shapes on all ranks
print(f"Rank {dist.get_rank()}: tensor.shape = {tensor.shape}")
dist.barrier()
Timeout Adjustment (for debugging only):
from areal.engine.core.distributed import patch_dist_group_timeout
from datetime import timedelta
patch_dist_group_timeout(timedelta(minutes=30))
Check DTensor Placements:
from torch.distributed.tensor import DTensor
if isinstance(param, DTensor):
print(f"Param {name}: placements={param.placements}, mesh={param.device_mesh}")
Verify Gradient Reduction:
for name, param in model.named_parameters():
if param.grad is not None:
print(f"Rank {dist.get_rank()}: {name} grad_sum = {param.grad.sum().item()}")
Check Memory Usage:
print(f"Rank {dist.get_rank()}: "
f"allocated={torch.cuda.memory_allocated()/1e9:.2f}GB, "
f"reserved={torch.cuda.memory_reserved()/1e9:.2f}GB")
Check FSDP Coverage:
for name, param in model.named_parameters():
is_dtensor = isinstance(param, DTensor)
print(f"{name}: is_dtensor={is_dtensor}, shape={param.shape}")
| Error | Cause | Solution |
|---|---|---|
NCCL WARN Cuda failure | GPU communication | Check NCCL version, GPU topology |
RuntimeError: Timed out | Rank synchronization | Increase timeout, check code paths |
Invalid device mesh | Mesh configuration | Verify world_size = dp * tp * cp |
| Variable | Purpose |
|---|---|
TORCH_DISTRIBUTED_DEBUG=DETAIL | Detailed distributed logging |
NCCL_DEBUG=INFO | NCCL communication logging |
NCCL_DEBUG_SUBSYS=ALL | All NCCL subsystems |
TORCH_LOGS="+dynamo,recompiles" | torch.compile logging |
TORCHDYNAMO_VERBOSE=1 | Dynamo verbose output |
CUDA_LAUNCH_BLOCKING=1 | Synchronous CUDA (slow, for debugging) |
# Install
pip install py-spy
# Dump call stack of hung process
py-spy dump --pid <PID>
# Dump all Python processes
pgrep -f python | xargs -I {} py-spy dump --pid {}
# Record flame graph
py-spy record -o profile.svg --pid <PID> --duration 30
def print_all_ranks(msg):
for r in range(dist.get_world_size()):
if dist.get_rank() == r:
print(f"[Rank {r}] {msg}")
dist.barrier()
def debug_mesh(parallel_dims):
mesh = parallel_dims.world_mesh
for dim_name in mesh.mesh_dim_names:
submesh = parallel_dims.get_mesh(dim_name)
if submesh:
print(f"Rank {dist.get_rank()}: {dim_name} size={submesh.size()}")
def check_tensor_consistency(tensor, name, group=None):
local_sum = tensor.sum().item()
tensor_sums = [None] * dist.get_world_size(group)
dist.all_gather_object(tensor_sums, local_sum, group=group)
if dist.get_rank() == 0 and len(set(tensor_sums)) > 1:
print(f"WARNING: {name} inconsistent: {tensor_sums}")
| Component | File |
|---|---|
| Parallel Dims | areal/experimental/models/archon/parallel_dims.py |
| Expert Parallel | areal/experimental/models/archon/expert_parallel.py |
| Ulysses (CP) | areal/experimental/models/archon/ulysses.py |
| FSDP/TP Apply | areal/experimental/models/archon/qwen2/infra/parallelize.py |