perf-torch-sync-free

# Writing Sync-Free PyTorch Code Sync-free code means the CPU continuously queues work to the GPU without waiting for GPU operations to complete. When host-device synchronizations are eliminated, the GPU works continuously without idle stalls. Every host-device synchronization ultimately calls one of three CUDA driver APIs that block the CPU thread: - `cuEventSynchronize` -- CPU waits until a specific GPU event completes - `cuStreamSynchronize` -- CPU waits until all work on a stream finishes - `cuCtxSynchronize` -- CPU waits until all work across all streams finishes ## When to Use Reach for this skill when you encounter: - **Triggers**: User wants to remove host-device synchronizations, eliminate CPU stalls from GPU waits, make code async/sync-free, remove `.item()` or `.cpu()` calls that block the CPU, or understand why specific PyTorch operations cause synchronization - **Symptoms**: Frequent `cudaStreamSynchronize` in nsys profiles, warnings from `torch.cuda.set_sync_debug_mode`, training throughput limited by CPU-GPU round-trips, `.item()` or `.cpu()` calls in hot loops - **Keywords**: "sync-free", "synchronization", ".item()", ".cpu()", "host-device sync", "eliminate syncs", "CPU stall", "non_blocking", "set_sync_debug_mode", "cudaStreamSynchronize", "cudaEventSynchronize", "remove syncs", "async GPU", "CPU waiting on GPU" Do NOT use this skill for: - Applying CUDA Graphs or reducing kernel launch overhead (use `perf-torch-cuda-graphs` instead) - Profiling GPU kernels, system timelines, or finding GPU idle time (use `perf-nsight-compute-analysis` or `perf-nsight-systems`) - Kernel optimization or code generation (use `kernel-triton-writing`) - Optimizing NCCL communication or distributed training collective operations - Reducing GPU memory usage or gradient checkpointing - General model compilation with `torch.compile` ## Requirements | Dependency | Version | Notes | |------------|---------|-------| | PyTorch | >=2.0 | With CUDA support | | NVIDIA GPU | Any | CUDA-capable | | Nsight Systems | Optional | For comprehensive sync detection via `nsys` | ## Workflow ### Step 1: Detect Synchronizations Use one or both methods to find sync points in the code. **Quick detection** -- PyTorch sync debug mode prints a warning with stack trace on every synchronization: ```python import torch # Enable at the start of the region you want to check torch.cuda.set_sync_debug_mode('warn') # prints warning + stack trace # torch.cuda.set_sync_debug_mode('error') # raises exception on sync # Run your training step / forward pass here train_step(model, batch) torch.cuda.set_sync_debug_mode(0) # disable ``` This mode only detects syncs going through PyTorch's wrapped `cuStreamSynchronize`. Third-party libraries calling CUDA sync APIs directly are not detected. **Comprehensive detection** -- Nsight Systems captures all sync calls including those from extensions and libraries: ```bash nsys profile --capture-range=cudaProfilerApi \ --python-sampling=true \ --backtrace=dwarf \ python your_script.py ``` In the Nsight Systems GUI, check the **CUDA API** timeline row and search for `cudaStreamSynchronize`, `cudaEventSynchronize`, or `cudaDeviceSynchronize`. The call stack panel shows which Python line triggered each sync. ### Step 2: Classify -- False vs True Dependencies After detecting syncs, classify each one before deciding how to fix it. **False dependencies** (avoidable) -- CPU does not actually need the GPU result. These can be eliminated without changing program logic: - Debug prints left in hot paths (`print(loss.item())`) - Unnecessary `.item()` calls for logging that could be deferred - Using `.cuda()` instead of `.to('cuda', non_blocking=True)` - Using `.type(torch.LongTensor)` instead of `.type(torch.long)` - Creating tensors from Python objects directly on CUDA **True dependencies** (require restructuring) -- CPU genuinely needs the GPU value to proceed: - **Control flow dependency**: `if loss.item() > threshold:` -- CPU branches on a GPU-computed value - **Dynamic memory allocation**: `output = x[mask]` -- output size depends on GPU computation - **CPU computation using GPU values**: computing statistics for logging, updating learning rates from metrics True dependencies require restructuring: move logic to GPU (`torch.where()`), delay to end of iteration, or accept that those parts stay outside any CUDA Graph capture region. ### Step 3: Eliminate Systematically Apply fixes in order of increasing difficulty. Start with easy wins. **1. Remove redundancy** -- Delete operations that do not need to happen: - Remove debug prints and logging from hot loops - Delete unnecessary `.item()` calls - Eliminate duplicate synchronizations **2. Use `non_blocking=True`** -- Make transfers async where CPU does not immediately use the result: ```python # Before (syncs) x_gpu = x_cpu.cuda() x_cpu = x_gpu.cpu() # After (async, no sync) x_gpu = x_cpu.to('cuda', non_blocking=True) x_cpu = x_gpu.to('cpu', non_blocking=True) # only if CPU does not use x_cpu immediately ``` Only use `non_blocking=True` for GPU-to-CPU when the CPU does not immediately read the result. Otherwise the CPU may operate on incomplete data. **3. Switch to sync-free API alternatives** -- See the Quick Reference Table below for a condensed mapping of common patterns. **4. Delay synchronization to end of iteration** -- Move logging and validation to after the optimizer step rather than mid-forward/backward: ```python # Before: sync mid-iteration loss = model(batch) print(f"Loss: {loss.item()}") # cuStreamSynchronize loss.backward() # After: delay to end of iteration loss = model(batch) loss.backward() optimizer.step() print(f"Loss: {loss.item()}") # sync is outside the hot path ``` **5. Coalesce multiple syncs into one** -- If you need several GPU values on CPU, gather them and transfer once: ```python # Before: 3 separate syncs loss_val = loss.item() # cuStreamSynchronize acc_val = accuracy.item() # cuStreamSynchronize gnorm_val = grad_norm.item() # cuStreamSynchronize # After: 1 sync metrics = torch.stack([loss, accuracy, grad_norm]) vals = metrics.cpu() # single cuStreamSynchronize loss_val, acc_val, gnorm_val = vals.tolist() ``` **6. Offload logic to GPU** -- Replace CPU-side logic with GPU-native ops: ```python # Before: CPU control flow (syncs) if loss.item() > threshold: result = a else: result = b # After: GPU-side selection (no sync) result = torch.where(loss > threshold, a, b) # Before: Python max (syncs) val = max(x_gpu[0, 0], x_gpu[0, 1]) # After: torch.max (no sync) val = torch.max(x_gpu[0, 0], x_gpu[0, 1]) ``` **7. Exclude unavoidable syncs from capture range** (last resort) -- If a sync cannot be eliminated, keep it outside the CUDA Graph capture region and graph only the sync-free sections. Partial graphing is better than no graphing. ### Step 4: Verify Re-run detection to confirm syncs are eliminated: ```python torch.cuda.set_sync_debug_mode('error') # will raise if any sync remains train_step(model, batch) torch.cuda.set_sync_debug_mode(0) ``` Or re-profile with Nsight Systems and confirm no `cudaStreamSynchronize` / `cudaEventSynchronize` / `cudaDeviceSynchronize` calls appear in the target region. ## Quick Reference Table | Sync-Inducing Pattern | Sync-Free Alternative | |----------------------|----------------------| | **Device Transfers** | | | `.cpu()` or `.to('cpu')` | `.to('cpu', non_blocking=True)` (fire-and-forget only) | | `.cuda()` or `.to('cuda')` | `.to('cuda', non_blocking=True)` | | `.type(torch.LongTensor)` | `.type(torch.long)` (dtype conversion, stays on GPU) | | **Tensor Creation** | | | `torch.tensor(obj, device='cuda')` | Create on CPU, then `.to('cuda', non_blocking=True)` | | `torch.tensor(0, device='cuda')` | `torch.zeros(1, device='cuda', dtype=...).squeeze()` | | `torch.as_tensor(arr, device='cuda')` | Create on CPU, then `.to('cuda', non_blocking=True)` | | `torch.cuda.BoolTensor(list)` | `torch.tensor(list, device='cpu').to('cuda', non_blocking=True)` | | **Control Flow** | | | `.item()` in conditionals | `torch.where()` or move outside critical region | | `if gpu_tensor:` | Keep logic on GPU with `torch.where()` | | Python `max(a, b)` on GPU tensors | `torch.max(a, b)` | | `torch.is_nonzero(t)` | Avoid; use GPU-side comparisons | | **Indexing** | | | `x_gpu[idx_cpu]` or `x_gpu[idx_list]` | `x_gpu[idx_gpu]` (keep indices on same device) | | `x_gpu[idx] = 0` (scalar assignment) | `x_gpu[idx] = zero_gpu` (GPU tensor value) | | `x[i:j]` with CUDA tensor bounds | `x[:, s]` with `s = torch.arange(i, j, device='cuda')` | | **Dynamic Shapes** | | | `x_gpu[mask_gpu]` (masked selection) | `torch.where(mask_gpu, x_gpu, 0)` (fixed shape) | | `torch.nonzero(mask)` | `torch.where()` or move outside critical region | | `torch.masked_select(x, mask)` | `torch.where(mask, x, 0)` | | `torch.unique(x)` | Avoid in hot path; precompute if possible | | `torch.repeat_interleave(x, r)` | Specify `output_size=N` if known | ## Finding More Information - **Tier 1 (this file)**: Workflow, classification, elimination strategies, and quick reference table - **Tier 2 (`references/sync-patterns.md`)**: Comprehensive pattern catalog with 9 categories, full code examples showing sync-inducing and sync-free versions, and the specific CUDA driver API triggered by each pattern