| name | warp-debug-gradients |
| description | Use to diagnose and fix incorrect gradients in differentiable Warp programs. Anything trained, optimized, calibrated, or fit through Warp kernels depends on wp.Tape gradients, so treat any misbehavior of such a workflow as a gradient problem until proven otherwise — use this when training diverges or NaNs, won't train at all, stalls or plateaus above the expected loss, converges to a wrong or biased answer, is worse than a reference implementation, works at small scale but fails at production scale, or fails a QA/validation recheck. Also for explicit symptoms — exploding, NaN/inf, zero, or subtly wrong gradients, suspected wp.Tape/backward issues, gradcheck failures — but users usually describe only the surface symptom ("the sim explodes", "the fit gets dragged toward outliers") without mentioning gradients: make that leap. Not for forward-only Warp work, build/install problems, or autograd issues in other frameworks without Warp. |
| license | Apache-2.0 |
| compatibility | Requires a working NVIDIA Warp installation (>= 1.13 minimum; >= 1.17 recommended for reliable verification — copy-adjoint accumulation, overwrite-warning call sites, read-flag lifetime, and gradcheck's restore_inputs changed in 1.17 and are version-caveated in the references; on older versions a fixed bug class still exists and some tools need workarounds). Diagnosis runs the user's reproduction, so a functioning device (CPU or CUDA) is needed. |
| metadata | {"author":"Warp Team <warp-python@nvidia.com>","version":"0.1.0","tags":["warp","autodiff","gradients","differentiable-simulation","debugging"],"upstream":"https://github.com/NVIDIA/warp"} |
Debugging Gradients in Warp
Gradient bugs in Warp are almost never math bugs. The forward simulation looks
perfectly healthy while the backward pass silently reads clobbered values,
skips arrays, or double-counts adjoints. Users routinely burn days tuning
physics knobs, loss functions, and assets when the real cause is a two-line
taping-pattern fix. Your job is to find that fix with evidence, not intuition.
The single most important discipline: measure before hypothesizing. It is
cheap for you to run a shrunk reproduction and compare autodiff against finite
differences. The way the gradient is wrong (its signature) prunes the
hypothesis space far faster than reading code ever will. Do not start
proposing fixes from code reading alone — plausible-looking diagnoses of
differentiability bugs are very often wrong, and an unverified "fix" that
happens to perturb the numbers wastes everyone's time.
When to Use This Skill
Anything trained, optimized, calibrated, or fit through Warp kernels flows
through wp.Tape gradients — so when such a workflow misbehaves, gradients
are the prime suspect even if the user never says the word. Activate on the
symptoms users actually report: training that diverges, NaNs, or does
nothing; loss that stalls or plateaus above where it should; fits that
converge to a wrong or biased answer or are worse than a reference
implementation; pipelines that work at small scale but fail at production
scale or fail a QA recheck. Also activate on explicit gradient symptoms —
exploding, NaN/inf, zero, or subtly wrong gradients,
wp.autograd.gradcheck failures, suspected wp.Tape/backward issues — and
when the user asks whether their gradients can be trusted.
Do not activate for forward-only Warp work (kernel authoring, rendering,
performance tuning), Warp build or installation problems, autograd questions
in other frameworks with no Warp involvement, or pure performance work on a
backward pass whose gradients the user has already validated.
The canonical background is Warp's own documentation — consult the relevant
section before diagnosing in its territory (online at
https://nvidia.github.io/warp/stable/; in a Warp source checkout the same content
is under docs/user_guide/; pip installs do not include it):
- The "Differentiability" guide — especially "Array Overwrites", "Debugging
Gradients", "Array Overwrite Tracking", and "Limitations and Workarounds"
(in-place math, component assignment, dynamic loops).
- The FAQ, section "Differentiation and Interoperability" — what state a
tape does and does not preserve, and checkpointing.
Prerequisites
Executing this skill assumes all of the following; if one is missing,
surface that to the user instead of improvising around it:
