Identify and eliminate host-device synchronizations in PyTorch code. Detects sync points (.item(), .cpu(), boolean indexing, torch.tensor on CUDA), classifies false vs true dependencies, provides sync-free alternatives. Triggers: sync-free, synchronization, .item(), .cpu(), host-device sync, eliminate syncs, CPU stall, non_blocking, set_sync_debug_mode, cudaStreamSynchronize, cudaEventSynchronize, remove syncs, async GPU.
tags
["synchronization","performance","pytorch"]
license
Apache-2.0
metadata
{"author":"NVIDIA Corporation"}
Writing Sync-Free PyTorch Code
Sync-free code means the CPU continuously queues work to the GPU without
waiting for GPU operations to complete. When host-device synchronizations
are eliminated, the GPU works continuously without idle stalls.
Every host-device synchronization ultimately calls one of three CUDA driver
APIs that block the CPU thread:
cuEventSynchronize -- CPU waits until a specific GPU event completes
cuStreamSynchronize -- CPU waits until all work on a stream finishes
cuCtxSynchronize -- CPU waits until all work across all streams finishes
When to Use
Reach for this skill when you encounter:
Triggers: User wants to remove host-device synchronizations, eliminate
CPU stalls from GPU waits, make code async/sync-free, remove .item() or
.cpu() calls that block the CPU, or understand why specific PyTorch
operations cause synchronization
Symptoms: Frequent cudaStreamSynchronize in nsys profiles,
warnings from torch.cuda.set_sync_debug_mode, training throughput
limited by CPU-GPU round-trips, .item() or .cpu() calls in hot loops
Applying CUDA Graphs or reducing kernel launch overhead (use
perf-torch-cuda-graphs instead)
Profiling GPU kernels, system timelines, or finding GPU idle time (use
perf-nsight-compute-analysis or perf-nsight-systems)
Kernel optimization or code generation (use kernel-triton-writing)
Optimizing NCCL communication or distributed training collective
operations
Reducing GPU memory usage or gradient checkpointing
General model compilation with torch.compile
Requirements
Dependency
Version
Notes
PyTorch
>=2.0
With CUDA support
NVIDIA GPU
Any
CUDA-capable
Nsight Systems
Optional
For comprehensive sync detection via nsys
Workflow
Step 1: Detect Synchronizations
Use one or both methods to find sync points in the code.
Quick detection -- PyTorch sync debug mode prints a warning with stack
trace on every synchronization:
import torch
# Enable at the start of the region you want to check
torch.cuda.set_sync_debug_mode('warn') # prints warning + stack trace# torch.cuda.set_sync_debug_mode('error') # raises exception on sync# Run your training step / forward pass here
train_step(model, batch)
torch.cuda.set_sync_debug_mode(0) # disable
This mode only detects syncs going through PyTorch's wrapped
cuStreamSynchronize. Third-party libraries calling CUDA sync APIs
directly are not detected.
Comprehensive detection -- Nsight Systems captures all sync calls
including those from extensions and libraries:
In the Nsight Systems GUI, check the CUDA API timeline row and search
for cudaStreamSynchronize, cudaEventSynchronize, or
cudaDeviceSynchronize. The call stack panel shows which Python line
triggered each sync.
Step 2: Classify -- False vs True Dependencies
After detecting syncs, classify each one before deciding how to fix it.
False dependencies (avoidable) -- CPU does not actually need the GPU
result. These can be eliminated without changing program logic:
Debug prints left in hot paths (print(loss.item()))
Unnecessary .item() calls for logging that could be deferred
Using .cuda() instead of .to('cuda', non_blocking=True)
Using .type(torch.LongTensor) instead of .type(torch.long)
Creating tensors from Python objects directly on CUDA
True dependencies (require restructuring) -- CPU genuinely needs the
GPU value to proceed:
Control flow dependency: if loss.item() > threshold: -- CPU
branches on a GPU-computed value
CPU computation using GPU values: computing statistics for logging,
updating learning rates from metrics
True dependencies require restructuring: move logic to GPU
(torch.where()), delay to end of iteration, or accept that those parts
stay outside any CUDA Graph capture region.
Step 3: Eliminate Systematically
Apply fixes in order of increasing difficulty. Start with easy wins.
1. Remove redundancy -- Delete operations that do not need to happen:
Remove debug prints and logging from hot loops
Delete unnecessary .item() calls
Eliminate duplicate synchronizations
2. Use non_blocking=True -- Make transfers async where CPU does not
immediately use the result:
# Before (syncs)
x_gpu = x_cpu.cuda()
x_cpu = x_gpu.cpu()
# After (async, no sync)
x_gpu = x_cpu.to('cuda', non_blocking=True)
x_cpu = x_gpu.to('cpu', non_blocking=True) # only if CPU does not use x_cpu immediately
Only use non_blocking=True for GPU-to-CPU when the CPU does not
immediately read the result. Otherwise the CPU may operate on incomplete
data.
3. Switch to sync-free API alternatives -- See the Quick Reference
Table below for a condensed mapping of common patterns.
4. Delay synchronization to end of iteration -- Move logging and
validation to after the optimizer step rather than mid-forward/backward:
# Before: sync mid-iteration
loss = model(batch)
print(f"Loss: {loss.item()}") # cuStreamSynchronize
loss.backward()
# After: delay to end of iteration
loss = model(batch)
loss.backward()
optimizer.step()
print(f"Loss: {loss.item()}") # sync is outside the hot path
5. Coalesce multiple syncs into one -- If you need several GPU values
on CPU, gather them and transfer once:
6. Offload logic to GPU -- Replace CPU-side logic with GPU-native ops:
# Before: CPU control flow (syncs)if loss.item() > threshold:
result = a
else:
result = b
# After: GPU-side selection (no sync)
result = torch.where(loss > threshold, a, b)
# Before: Python max (syncs)
val = max(x_gpu[0, 0], x_gpu[0, 1])
# After: torch.max (no sync)
val = torch.max(x_gpu[0, 0], x_gpu[0, 1])
7. Exclude unavoidable syncs from capture range (last resort) -- If a
sync cannot be eliminated, keep it outside the CUDA Graph capture region
and graph only the sync-free sections. Partial graphing is better than no
graphing.
Step 4: Verify
Re-run detection to confirm syncs are eliminated:
torch.cuda.set_sync_debug_mode('error') # will raise if any sync remains
train_step(model, batch)
torch.cuda.set_sync_debug_mode(0)
Or re-profile with Nsight Systems and confirm no cudaStreamSynchronize /
cudaEventSynchronize / cudaDeviceSynchronize calls appear in the
target region.
Tier 2 (references/sync-patterns.md): Comprehensive pattern catalog
with 9 categories, full code examples showing sync-inducing and sync-free
versions, and the specific CUDA driver API triggered by each pattern
