3dgs-training-debugger

Diagnose and fix 3DGS training-time failures: NaN losses, OOM crashes, divergent optimization, floater artifacts, densification failures, hyperparameter sensitivity. Covers runtime debugging for vanilla 3DGS and 50+ novel methods (deformable, MoE, physics-based, feed-forward). Detects 60 runtime failure patterns. Use when: 3DGS training crashes or produces poor results, loss is NaN/Inf, VRAM exhaustion, Gaussians explode or vanish, densification not working, convergence stalls, 训练调试/显存溢出/训练发散/浮点伪影.

Ir para a instalação

Repositório: jaccen/Awesome-Gaussian-Skills
Última atividade na origem: 7 de agosto de 2026 às 10:46
Idioma detectado do SKILL.md: inglês
Estrelas: 143
Forks: 10

Opções de instalação

Por padrão, está selecionado o prompt que primeiro revisa a origem. Você pode mudar para um comando direto ou baixar uma cópia local.

Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.

Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.

npx skills add https://github.com/jaccen/Awesome-Gaussian-Skills --skill 3dgs-training-debugger

O comando permanece em uma só linha. Role horizontalmente para revisá-lo antes de copiar.

Prefere uma cópia local? Baixe os arquivos disponíveis atualmente no SkillsMP.

Revise os arquivos de origem

Leia o SKILL.md e os arquivos complementares exibidos pelo SkillsMP antes de decidir se vai instalar.

Explorador de arquivos

4 arquivos

SKILL.md

Instruções da origem · Visualização somente leitura

Voltar à pesquisa Voltar à instalação

name	3dgs-training-debugger
description	Diagnose and fix 3DGS training-time failures: NaN losses, OOM crashes, divergent optimization, floater artifacts, densification failures, hyperparameter sensitivity. Covers runtime debugging for vanilla 3DGS and 50+ novel methods (deformable, MoE, physics-based, feed-forward). Detects 60 runtime failure patterns. Use when: 3DGS training crashes or produces poor results, loss is NaN/Inf, VRAM exhaustion, Gaussians explode or vanish, densification not working, convergence stalls, 训练调试/显存溢出/训练发散/浮点伪影.
license	Apache-2.0
user-invocable	true
metadata	{"version":"1.0.0","author":"jaccen","tags":["3dgs","gaussian-splatting","training","debugging","oom","convergence","hyperparameters","distributed"],"when_to_use":["3DGS training crashes with OOM or CUDA error","Training loss becomes NaN or Inf","Gaussian count explodes or vanishes during training","Densification is not triggering or over-triggering","Training converges slowly or plateaus prematurely","Visual artifacts appear (floaters, blur, ghosting) after training","Checkpoint resume produces different results","Distributed training has inconsistent gradients or dead workers","Novel method (deformable/MoE/physics) diverges","训练调试 / 显存溢出 / 训练发散 / 浮点伪影 / 密度控制异常"]}

3DGS Training Debugger

You are a senior 3DGS engineer who has trained hundreds of Gaussian Splatting models across vanilla 3DGS, deformable GS, feed-forward GS, SLAM-GS, and physics-based GS pipelines. Diagnose and fix training-time failures systematically.

SYMPTOM: Training crashed or NaN │ ├── NaN/Inf in loss? │ ├── Check gradient norms → extremely large? │ │ └── Possible: learning rate too high, gradient explosion │ │ → Reduce lr to 1/10, add gradient clipping (max_norm=1.0) │ ├── Check after ADC cycle → NaN appears right after densification? │ │ └── Possible: NaN from clone/split, new Gaussian has bad scale/opacity │ │ → Check scale clamping, opacity init values │ ├── NaN from iteration 0? │ │ └── Possible: bad initialization (zero covariance, SfM failure) │ │ → Check point cloud, add covariance regularization │ └── NaN in novel method (deformable/MoE)? │ └── See Section 9: Novel Method Stability │ ├── CUDA OOM? │ ├── During training (non-ADC)? │ │ ├── Gaussian count reasonable but still OOM? │ │ │ └── Possible: image resolution too high, batch size, SH degree │ │ │ → Reduce image res by 2x, reduce SH to degree 1 │ │ └── Gaussian count exploding? │ │ └── Possible: densification over-triggering │ │ → See Section 3: OOM & Memory Management │ └── During ADC (densification)? │ └── Possible: temporary spike from clone/split │ → Reduce ADC batch size, or move ADC to CPU │ ├── Training runs but quality is poor (low PSNR)? │ ├── Gaussian count too low? │ │ └── Possible: densification thresholds too strict, pruning too aggressive │ │ → Lower grad_threshold, raise prune_threshold │ ├── Gaussian count normal but artifacts? │ │ └── See Section 4: Artifact Diagnosis │ ├── Convergence stalled early? │ │ └── See Section 6: Convergence Analysis │ └── Specific views are bad? │ └── Possible: training/test view selection issue, SfM sparse in that area │ ├── Training runs but visual artifacts? │ ├── Floaters (small isolated Gaussians)? │ │ └── See Pattern FP-01 in references │ ├── Blur / over-smoothing? │ │ └── See Pattern FP-02 │ ├── Ghosting / duplicate geometry? │ │ └── See Pattern FP-03 │ ├── Holes / missing regions? │ │ └── See Pattern FP-04 │ └── Color bleeding / SH artifacts? │ └── See Pattern FP-05 │ └── Checkpoint resume gives different results? └── See Section 8: Checkpoint & Resume

Metric	Expected Behavior	Alert Threshold	Log Frequency
L1 loss	Decreasing, minor oscillation	Increase > 20% over 500 iters	Every 50 iters
SSIM loss	Decreasing smoothly	Stagnant for 1000+ iters	Every 100 iters
Total loss	Decreasing, plateau ~70-80% of training	NaN, Inf, or sudden spike	Every 50 iters
PSNR (eval)	Increasing, plateau near end	Drop > 2dB between evals	Every 1000 iters
Gaussian count	Growth phase (0-15k), then stable	Explosive growth (>10x) or vanishing	Every 500 iters
VRAM usage	Stable with minor fluctuation during ADC	> 90% of total VRAM	Every 100 iters
Gradient norms	Stable, < 1.0 typically	> 10.0 or exactly 0.0	Every 100 iters
Learning rate	Following schedule (warmup → cosine decay)	Unexpected reset or spike	Every 500 iters
Active Gaussians	Growth then pruning equilibrium	All pruned (count → 0)	Every ADC cycle
ADC trigger count	Periodic (every ~100 iters)	Never triggers or triggers every iter	Every ADC cycle

Priority	Strategy	VRAM Savings	Quality Impact	Implementation
1	Reduce image resolution (2x downsample)	50-75% raster VRAM	Minor PSNR drop (~0.5-1dB)	`--data_factor 2`
2	Lower SH degree (3→1)	~40% model VRAM	Slight view-dependent color loss	`--sh_degree 1`
3	Gradient checkpointing on rasterizer	30-40% gradient VRAM	~10% slower training	Custom backward pass
4	Reduce ADC frequency (100→200 iters)	Reduces ADC spike frequency	Slower densification	`--densify_interval 200`
5	CPU-offload optimizer states	40% total VRAM	~30% slower (PCIe transfer)	FSDP/DeepSpeed
6	Mixed precision (FP16/BF16) training	30-50% total VRAM	Risk of numerical instability	`torch.cuda.amp`
7	Streaming image loading (not all in VRAM)	Major for large datasets	No quality impact	Custom data loader
8	Prune far-away Gaussians aggressively	Reduces model VRAM	May lose background detail	Custom prune criterion

Scenario	Typical Cause	Fix
OOM at iter ~500 (first ADC)	Sudden Gaussian count jump	Pre-allocate buffer for 5x initial count
OOM only on specific scenes	High-detail scenes grow more Gaussians	Scene-adaptive resolution reduction
OOM after checkpoint resume	Optimizer state not saved/loaded	Save full optimizer state in checkpoint
OOM on multi-GPU	All-reduce buffer too large	Gradient bucketing, overlap comm/compute
OOM with novel method	Extra params (deformation, MLP)	Profile each component separately

Artifact	Visual Symptom	Most Likely Training Cause	Diagnostic Action
Floaters	Small bright/dark blobs floating in space	Insufficient opacity pruning; ADC cloning noise	Check prune_opacity threshold; check if ADC ran after iter 15k
Blur	Overall soft, lacks high-freq detail	SH degree too low; low-resolution training images	Increase SH to 3; check `--data_factor`
Over-smoothing	PSNR OK but LPIPS bad, looks "flat"	L1+SSIM loss too weighted to L1; insufficient iterations	Increase SSIM weight (λ_dssim > 0.2)
Ghosting	Duplicate/semi-transparent geometry	Clone in wrong direction; scale gradient sign error	Check ADC clone position offset; verify gradient direction
Holes	Black/empty regions in reconstruction	Over-aggressive pruning; SfM sparse in that region	Raise prune threshold; add points in sparse areas
Color bleeding	Color from one surface leaks to another	SH coefficient overflow; insufficient view coverage	Clamp SH values; check training camera distribution
Stretching	Elongated Gaussian streaks	Scale not clamped; bad covariance projection	Verify `scale_activation` clamping (max 0.1-10.0)
Popping	View-dependent flickering between views	SH degree too high with sparse views; opacity reset	Reduce SH degree; increase opacity reset iterations
Z-fighting	Flickering on overlapping surfaces	Near-duplicate Gaussians at same depth	Add uniqueness in clone; increase prune threshold
Dark scene	Overall too dark / underexposed	Background color set to black; insufficient training	Set background to white or random; train longer

Parameter	Default	Range	Effect of Increase	Effect of Decrease
`position_lr`	0.00016	1e-5 to 1e-2	Faster convergence, risk of explosion	Slower, more stable
`feature_lr`	0.0025	1e-4 to 1e-1	Faster SH convergence	Slower color
`opacity_lr`	0.05	1e-3 to 0.2	Faster opacity adaptation	Slower prune response
`scaling_lr`	0.005	1e-4 to 0.05	Faster scale adaptation	More rigid geometry
`rotation_lr`	0.001	1e-5 to 0.01	Faster rotation adaptation	More rigid orientation
`densify_grad_threshold`	0.0002	1e-5 to 1e-2	More Gaussians (sensitive)	Fewer Gaussians
`densify_interval`	100	50-500	Less frequent densification	More frequent
`densify_until_iter`	15000	5000-30000	Longer growth phase	Earlier freeze
`prune_opacity_threshold`	0.005	0.001-0.05	More aggressive pruning (fewer floaters)	More Gaussians (risk floaters)
`opacity_reset_interval`	3000	1000-10000	More frequent resets (less view-dep overfit)	More stable opacities
`sh_degree`	3	0-4	Better view-dependent color	Less VRAM
`lambda_dssim`	0.2	0-1	More structural similarity	More pixel-level accuracy

Problem	First Adjustment	Second Adjustment	Last Resort
Low PSNR	Lower `densify_grad_threshold` (more Gaussians)	Increase training iterations	Lower image resolution
OOM	Lower `sh_degree`	Reduce image resolution	Decrease `densify_until_iter`
Floaters	Raise `prune_opacity_threshold`	Increase `opacity_reset_interval`	Post-train prune
Blur	Increase `sh_degree`	Increase `lambda_dssim`	Higher resolution images
Slow convergence	Increase `position_lr`	Increase `densify_interval`	Fewer total iters (accept lower quality)
Divergence	Decrease all LRs by 10x	Add gradient clipping	Reduce batch complexity
Too many Gaussians	Raise `densify_grad_threshold`	Lower `densify_until_iter`	Aggressive pruning

Failure Mode	Symptom	Root Cause	Fix
Premature plateau	PSNR stops improving by iter 5,000	Densification frozen too early; lr too low	Increase `densify_until_iter`; raise lr
Never converges	Loss oscillates, PSNR ~20 at iter 30k	Learning rate too high; bad initialization	Reduce lr 10x; check SfM point cloud
Train-good/test-bad	High train PSNR, low test PSNR	Overfitting; insufficient camera coverage	More cameras; early stopping; regularization
Sudden regression	PSNR drops dramatically mid-training	Gradient explosion; bad ADC clone; data corruption	Check gradient norms; add clipping; verify data
Asymmetric convergence	Some views perfect, others terrible	SfM sparse in some regions; uneven camera distribution	Add cameras; increase densification in sparse areas
Late-stage degradation	PSNR peaks then declines	Overfitting SH; opacity over-adaptation	Early stopping at peak; reduce `opacity_lr`

Strategy	Description	When to Use	Pitfalls
Data Parallel (DDP)	Each GPU trains full model on different image batch	Standard for large datasets	All-reduce bottleneck with high Gaussian count; requires gradient sync
Model Parallel	Split Gaussians across GPUs	When single GPU VRAM insufficient	Load imbalance; complex rasterization coordination
Pipeline Parallel	Split training stages across GPUs	Rare for 3DGS	Not well-supported by rasterization kernels
FSDP	Shard optimizer states + gradients	Very large Gaussian counts	Overhead for moderate counts; CPU offload needed

Bug ID	Symptom	Cause	Fix
DT-01	Loss diverges on rank 0 only	Gradient sync issue; non-deterministic ADC	Use `torch.distributed.barrier()` before ADC
DT-02	Different Gaussians on different ranks	Densification not synchronized	Broadcast Gaussian count/positions after ADC
DT-03	Dead worker (hangs at all-reduce)	One GPU OOM; NCCL timeout	Monitor per-GPU VRAM; add NCCL timeout config
DT-04	Slower than single-GPU	All-reduce dominates compute	Use gradient bucketing; overlap comm/compute
DT-05	Checkpoint loads on 1 GPU, fails on multi	State dict has single-device tensors	Use `map_location` + DDP-aware state dict unwrap
DT-06	Non-reproducible results across runs	Non-deterministic cuDNL; random ADC ordering	Set seeds; use `torch.use_deterministic_algorithms(True)`

Issue	Symptom	Cause	Fix
Resume diverges	Loss jumps or NaN after resume	Optimizer state not saved/loaded	Always save + restore optimizer state_dict
Wrong Gaussian count	Gaussians mismatch on resume	ADC ran between save and resume	Save AFTER ADC cycle, not during
Shape mismatch	Tensor size error on load	Pruning changed Gaussian count	Save count explicitly; handle add/remove
SH degree mismatch	Feature dim error	SH degree auto-incremented during training	Save active_sh_degree; restore it
Scale/rotation mismatch	Bad rendering after resume	Activation functions applied during save	Save pre-activation values (_scaling, _rotation)
Novel method state lost	Deformation MLP reset on resume	Extra module state not in checkpoint	Include all module state_dicts in checkpoint

Method Family	Common Stability Issue	Root Cause	Mitigation
Deformable GS (4DGS, Deformable-3DGS)	Deformation MLP outputs NaN	Unconstrained MLP output; large gradients through time	Add tanh activation on output; gradient clip; warmup with frozen base
Feed-forward GS (pixelSplat, LRM-based)	Instability with few training images	Model predicts Gaussians from sparse views	More training iterations; 2D feature regularization
MoE-GS	Expert collapse (all routing to 1 expert)	Router imbalance; load balancing loss weight too low	Increase load-balancing loss; add router z-loss
Physics-based GS (PhysGaussian, Springs)	Physics simulation diverges	Large time step; unstable integrator	Reduce dt; use semi-implicit Euler; add damping
SLAM-GS	Drift accumulation over time	Incremental map update without global optimization	Periodic global BA; keyframe-based adjustment
Compression GS	Quality collapse after pruning	Pruned critical Gaussians	Importance-aware pruning; fine-tune after prune
GaussianGrasper / Embodied	Grasp success drops during training	Sim-to-real gap amplifies	Domain randomization; curriculum learning
PBR / Material GS	Material decomposition unstable	Joint optimization of geometry + material under-determined	Stage training: geometry first, then material
City-scale / Large-scene	OOM or spatial discontinuity	Too many Gaussians in one scene	Block-wise training; LOD hierarchy
GaussTrace / Provenance	Provenance tags mismatch after ADC	Clone/split not propagating tags	Custom ADC that preserves provenance metadata

Category	Count	Examples
Initialization failures (IF)	6	SfM sparse init, zero covariance, scale explosion
Densification failures (DF)	8	Over/under-triggering, clone direction error, split scale error
Optimization failures (OF)	7	LR explosion, gradient vanishing, loss masking error
Memory failures (MF)	6	OOM at ADC, VRAM fragmentation, optimizer state bloat
Convergence failures (CF)	7	Premature plateau, oscillation, test regression, asymmetry
Artisanal artifacts (AF)	8	Floaters, blur, ghosting, holes, color bleed
Multi-GPU failures (MF2)	6	Gradient sync, dead worker, desync ADC, NCCL timeout
Novel method failures (NF)	12+	Deformable NaN, MoE collapse, physics divergence, SLAM drift
Total	60+	See `references/runtime-bug-patterns.md`

3dgs-training-debugger

Informações da origem

Opções de instalação

Revise os arquivos de origem

Mais deste repositório

Mais deste repositório

3DGS Training Debugger

3DGS Training Debugger

Capabilities

Relationship to Other Skills

Section 1: Training Monitoring Checklist

1.1 What to Monitor During Training

1.2 Healthy Training Signature (Mip-NeRF 360, RTX 4090)

Section 2: Failure Diagnosis Decision Tree

Section 3: OOM & Memory Management

3.1 VRAM Estimation Formula

3.2 OOM Mitigation Strategies (Priority Order)

3.3 Common OOM Scenarios

Section 4: Artifact Diagnosis Catalog

4.1 Visual Artifact → Training Cause Mapping

4.2 Artifact-to-Iteration Diagnosis

Section 5: Hyperparameter Tuning Guide

5.1 Core Hyperparameters

5.2 Tuning Decision Matrix

Section 6: Convergence Analysis

6.1 Expected Convergence Phases

6.2 Convergence Failure Modes

Section 7: Distributed & Multi-GPU Training

7.1 Multi-GPU Strategies for 3DGS

7.2 Common Distributed Training Bugs

7.3 Multi-GPU Setup Checklist

Section 8: Checkpoint & Resume

8.1 What to Save in a Checkpoint

8.2 Common Checkpoint Issues

Section 9: Novel Method Training Stability

9.1 Method-Specific Stability Issues

9.2 General Novel Method Debugging Protocol

Section 10: Training Runtime Bug Patterns

Pattern Categories

Self-Check Loop

SC-1: Symptom Verification

SC-2: Diagnostic Completeness

SC-3: Fix Verification

Red Lines

Related Skills

Guardrail: Do Not Apply From Memory