renderdoc-gpu-debug

name

renderdoc-gpu-debug

description

GPU frame debugging with RenderDoc via rdc-cli. Use this skill when the user mentions: GPU debugging, .rdc files, RenderDoc, shader issues, pipeline state, rendering artifacts, frame capture, shadow problems, visual glitches, draw calls, render targets, pixel history, texture inspection, depth buffer, blend state, rasterizer state, vertex shader, fragment shader, pixel shader, compute shader, mesh output, GPU performance, overdraw, bandwidth, shadow map, PCF filtering, screen-space artifacts, z-fighting, alpha blending, stencil, MSAA, "why does this look wrong", "debug this pixel", "what's drawing here", "capture a frame", "inspect the pipeline", "export render target", vulkan debugging, D3D debugging, GL debugging. DO NOT use for: CSS rendering, React rendering, server-side rendering, HTML layout, browser DevTools, web performance, canvas 2D, SVG rendering.

RenderDoc GPU Debugging Skill

Overview

This skill enables GPU frame capture, inspection, and debugging using rdc-cli, a 66-command CLI wrapping RenderDoc's Python API. It works with Vulkan, D3D11, D3D12, and OpenGL applications.

Prerequisites

Before any GPU debugging, verify the environment:

rdc doctor

All checks should pass. If rdc doctor fails, check:

• RENDERDOC_PYTHON_PATH is set to the directory containing renderdoc.pyd
• renderdoc.dll is in the same directory as renderdoc.pyd
• renderdoccmd.exe is on PATH or in the module directory

1. Session Lifecycle

Every inspection session follows open-work-close:

rdc open path/to/capture.rdc   # Start daemon, load capture
# ... inspection commands ...
rdc close                       # Release resources, stop daemon

Check session state with rdc status. Only one capture can be open per session (use --session name for parallel sessions).

IMPORTANT: Always close sessions when done. Leaked daemon processes consume GPU memory.

2. Capture Workflow

Capture from an executable

rdc capture -o output.rdc -- /path/to/app [app-args...]

Key options:

• -o, --output: Output .rdc file path
• --frame N: Capture frame N (default: first presented frame)
• --timeout S: Capture timeout in seconds (default: 60)
• --api-validation: Enable API validation layer
• --ref-all-resources: Reference all resources (larger capture, more complete)
• --wait-for-exit: Wait for the app to exit after capture

CWD matters

Many applications resolve assets relative to CWD. Always set CWD appropriately:

cd /path/to/app/root && rdc capture -o /output/path/capture.rdc -- ./app

For lux-playground specifically:

cd D:/shaderlang && python D:/renderdoc/capture_frame.py

Vulkan layer requirement

For Vulkan apps, the RenderDoc Vulkan layer must be registered:

• Registry: HKCU\SOFTWARE\Khronos\Vulkan\ImplicitLayers must contain D:\renderdoc\module\renderdoc.json (DWORD 0)
• Environment: ENABLE_VULKAN_RENDERDOC_CAPTURE=1 must be set

If headless capture fails

Headless Vulkan apps (no swapchain / no vkQueuePresentKHR) cannot be captured by RenderDoc since frame boundaries are defined by present calls. Options:

1. Use --interactive mode if the app supports it (opens a window with a swapchain)
2. Use the Python API via capture_frame.py for reliable programmatic capture
3. Use --trigger mode (inject without auto-capture, then rdc capture-trigger)
4. Fall back to renderdoccmd capture directly

3. Frame Exploration

After opening a capture, start with high-level overview:

rdc info --json          # API, GPU, driver, resolution, frame number
rdc stats --json         # Per-pass breakdown, top draws, largest resources
rdc passes               # List render passes (debug markers / implicit passes)
rdc draws --limit 20     # First 20 draw calls
rdc count draws          # Total draw call count

Navigate by pass

rdc draws --pass "Shadow Pass" --json    # Draws in a specific pass
rdc draws --pass "GBuffer" --limit 10    # First 10 GBuffer draws

Navigate by event

rdc events --limit 50                    # All API events (not just draws)
rdc events --type DrawIndexed --json     # Filter by event type
rdc event 42 --json                      # Detail for one event

VFS browsing

The virtual filesystem provides a structured view of the capture:

rdc ls /                      # Root directories
rdc ls /textures -l           # List textures with metadata
rdc tree /pipelines --depth 2 # Pipeline tree
rdc cat /events/42            # Event detail

4. Pipeline State Inspection

Get full pipeline state at any draw call:

rdc pipeline EID --json              # Full pipeline state
rdc pipeline EID vs --json           # Vertex shader section only
rdc pipeline EID ps --json           # Pixel/fragment shader section only
rdc pipeline EID om --json           # Output merger (blend, depth, stencil)
rdc pipeline EID rs --json           # Rasterizer state (culling, viewport)
rdc pipeline EID ia --json           # Input assembler (vertex buffers, topology)
rdc pipeline EID ds --json           # Depth-stencil state

Bound resources

rdc bindings EID --json              # All bindings for all stages
rdc bindings EID --set 0 --json      # Descriptor set 0 only

5. Shader Inspection

View shader metadata and source

rdc shader EID vs --json             # Vertex shader metadata
rdc shader EID ps --json             # Fragment/pixel shader metadata
rdc shader EID ps --source           # Debug source code (if available)
rdc shader EID ps --reflect --json   # Reflection: inputs, outputs, cbuffers
rdc shader EID ps --constants --json # Constant buffer values at this draw

Disassembly targets

rdc shader EID ps --targets          # List available disassembly formats
rdc shader EID ps --target spirv     # SPIR-V disassembly
rdc shader EID ps --target glsl      # GLSL (cross-compiled, if available)

Search shaders

rdc search "shadow" --json           # Search all shader disassembly for "shadow"
rdc search "main" --stage ps         # Search only pixel shaders
rdc shaders --stage ps --json        # List all unique pixel shaders
rdc shader-map --json                # EID-to-shader mapping

6. Visual Inspection: Export-View-Analyze

This is the core pattern for visual debugging. Always export to PNG, then use Codex's Read tool to view the image.

Export render targets

rdc rt EID -o D:/renderdoc/captures/analysis/render_target.png
rdc rt EID --target 1 -o D:/renderdoc/captures/analysis/rt_target1.png  # MRT target 1

Export textures

rdc texture RESID -o D:/renderdoc/captures/analysis/texture.png
rdc texture RESID --mip 2 -o D:/renderdoc/captures/analysis/mip2.png

Export thumbnail

rdc thumbnail -o D:/renderdoc/captures/analysis/thumb.png

View exported images

After exporting a PNG, use the Read tool to view it (Codex is multimodal):

Read tool: D:/renderdoc/captures/analysis/render_target.png

Do NOT use cat or bash to view images. The Read tool renders them visually.

Analyze

Correlate what you see in the image with pipeline state data:

1. Export the render target at the suspicious draw
2. View it with the Read tool
3. Check pipeline state (rdc pipeline EID --json)
4. Check shader constants (rdc shader EID ps --constants --json)
5. Form hypothesis and verify

7. Pixel Debugging

Pixel history

Find all draws that wrote to a pixel:

rdc pixel X Y --json                 # Full pixel history at current event
rdc pixel X Y EID --json             # Pixel history up to EID

Pick pixel color

Read the current color at a pixel:

rdc pick-pixel X Y --json            # Color at (X,Y) from current render target
rdc pick-pixel X Y EID --json        # Color at specific event

Debug pixel shader

Step through shader execution for a specific pixel:

rdc debug pixel EID X Y --json           # Shader inputs/outputs summary
rdc debug pixel EID X Y --trace          # Full execution trace (every line)
rdc debug pixel EID X Y --dump-at 42     # Variable snapshot at line 42

Debug vertex shader

rdc debug vertex EID VTXID --json        # Vertex shader debug for vertex ID
rdc debug vertex EID VTXID --trace       # Full trace

Debug compute shader

rdc debug thread EID GX GY GZ TX TY TZ --json   # Compute thread debug

8. Shader Edit-Replay

Modify shaders without recompiling the application:

# 1. Check available encodings
rdc shader-encodings --json

# 2. Extract shader source
rdc shader EID ps --source -o D:/renderdoc/captures/analysis/shader.frag

# 3. Edit the shader (use Edit tool)

# 4. Build the modified shader
rdc shader-build D:/renderdoc/captures/analysis/shader.frag --encoding GLSL --stage ps --json

# 5. Replace (use the shader_id from build output)
rdc shader-replace EID ps --with SHADER_ID --json

# 6. Export to verify the change
rdc rt EID -o D:/renderdoc/captures/analysis/after_edit.png

# 7. Restore when done
rdc shader-restore EID ps
# or: rdc shader-restore-all

9. Frame Comparison

Compare two captures:

# Quick summary of differences
rdc diff capture_a.rdc capture_b.rdc --shortstat

# Detailed comparisons
rdc diff capture_a.rdc capture_b.rdc --draws --json       # Draw call differences
rdc diff capture_a.rdc capture_b.rdc --resources --json   # Resource differences
rdc diff capture_a.rdc capture_b.rdc --passes --json      # Pass structure differences
rdc diff capture_a.rdc capture_b.rdc --framebuffer --json  # Final framebuffer diff

# Visual diff
rdc diff capture_a.rdc capture_b.rdc --framebuffer --diff-output D:/renderdoc/captures/analysis/diff.png

10. Debugging Recipes

Recipe: Object is invisible

# 1. Find the draw call that should render the object
rdc draws --json | jq '.[] | select(.name | contains("ObjectName"))'
# or search by pass:
rdc draws --pass "Main Pass" --json

# 2. Check if it's being culled
rdc pipeline EID rs --json   # Look at CullMode, FrontFace

# 3. Check depth state
rdc pipeline EID ds --json   # DepthEnable, DepthFunc, DepthWriteMask

# 4. Check blend state (maybe alpha is 0)
rdc pipeline EID om --json   # BlendEnable, SrcBlend, DestBlend

# 5. Check vertex transform
rdc debug vertex EID 0 --json   # Is the position off-screen or behind camera?

# 6. Check if the draw is even issuing primitives
rdc draw EID --json   # VertexCount, InstanceCount, IndexCount

Recipe: Colors are wrong

# 1. Export the render target to see what's there
rdc rt EID -o D:/renderdoc/captures/analysis/wrong_color.png

# 2. Pick the problematic pixel
rdc pick-pixel X Y EID --json

# 3. Check texture bindings — is the right texture bound?
rdc bindings EID --json

# 4. Export the bound texture
rdc texture RESID -o D:/renderdoc/captures/analysis/bound_texture.png

# 5. Check shader constants — wrong material colors?
rdc shader EID ps --constants --json

# 6. Check blend state — additive when it should be alpha?
rdc pipeline EID om --json

# 7. Debug the pixel shader to trace the calculation
rdc debug pixel EID X Y --trace

Recipe: Shadows are broken

# 1. Find the shadow pass
rdc passes --json
rdc draws --pass "Shadow*" --json

# 2. Export the shadow map
SHADOW_EID=$(rdc draws --pass "Shadow*" -q | tail -1)
rdc rt $SHADOW_EID -o D:/renderdoc/captures/analysis/shadow_map.png

# 3. Check shadow map resolution (is it too small?)
rdc bindings $SHADOW_EID --json   # Look at render target dimensions

# 4. Check depth bias
rdc pipeline $SHADOW_EID rs --json   # DepthBias, SlopeScaledDepthBias

# 5. Find the lighting pass that reads the shadow map
rdc draws --pass "Raster*" --json
LIGHT_EID=$(rdc draws --pass "Raster*" -q | head -1)

# 6. Check how the shadow map is sampled
rdc shader $LIGHT_EID ps --source   # Look for shadow sampling code
rdc shader $LIGHT_EID ps --constants --json   # Light matrices, bias values

# 7. Debug a shadowed pixel
rdc debug pixel $LIGHT_EID X Y --trace

Recipe: Performance is bad

# 1. Get frame overview
rdc stats --json

# 2. Count draws per pass
rdc passes --json   # Look for passes with excessive draw counts

# 3. Look for redundant state changes
rdc events --limit 500 --json | jq 'group_by(.type) | map({type: .[0].type, count: length}) | sort_by(-.count)'

# 4. Check for large resources
rdc resources --sort size --json | jq '.[-10:]'   # Top 10 largest resources

# 5. Look for overdraw (if GPU counters available)
rdc counters --list
rdc counters --name "overdraw" --json

# 6. Export with wireframe overlay to visualize overdraw
rdc rt EID --overlay wireframe -o D:/renderdoc/captures/analysis/wireframe.png

Recipe: What changed between two frames

# Quick diff
rdc diff before.rdc after.rdc --shortstat

# Detailed diffs
rdc diff before.rdc after.rdc --draws --json
rdc diff before.rdc after.rdc --framebuffer --diff-output D:/renderdoc/captures/analysis/frame_diff.png

# Compare pipeline state at specific draw
rdc diff before.rdc after.rdc --pipeline EID --json

Recipe: Debug this pixel

# 1. Get pixel history — which draws touched this pixel?
rdc pixel X Y --json

# 2. Pick the draw that produced the final color
# (usually the last non-failing entry in pixel history)

# 3. Get the pixel's current value
rdc pick-pixel X Y EID --json

# 4. Debug the shader at that pixel
rdc debug pixel EID X Y --json

# 5. Get full execution trace if needed
rdc debug pixel EID X Y --trace

# 6. Check variable values at a specific shader line
rdc debug pixel EID X Y --dump-at LINE_NUMBER

11. Output Size Management

GPU captures can produce enormous output. Follow these rules:

1. Always use --limit for exploration: rdc draws --limit 20, rdc events --limit 50
2. Filter by pass: rdc draws --pass "Shadow Pass" instead of all draws
3. Use -q for ID lists: rdc draws -q returns only EIDs, not full details
4. Use --json selectively: JSON is verbose; use TSV default for scanning
5. Pipeline sections: rdc pipeline EID vs instead of full rdc pipeline EID
6. Limit search results: rdc search "pattern" --limit 10

Size estimates

• rdc info: ~20 lines
• rdc draws --limit 20: ~25 lines
• rdc pipeline EID --json: ~200-500 lines (full), ~50 lines (per section)
• rdc shader EID ps --source: ~50-500 lines depending on shader
• rdc debug pixel EID X Y --trace: ~100-1000 lines depending on shader complexity

12. Error Handling

rdc not found

pip install rdc-cli

rdc doctor fails

Check RENDERDOC_PYTHON_PATH points to directory containing renderdoc.pyd and renderdoc.dll.

Capture fails with "no swapchain"

The app may not present frames in the expected way. Try:

rdc capture --trigger -- /path/to/app   # Inject-only mode
# Then manually trigger: rdc capture-trigger

Daemon not responding

rdc status          # Check if daemon is alive
rdc close           # Force close
rdc open capture.rdc  # Re-open

GPU feature not supported

Some features (GPU counters, pixel history) depend on the GPU and driver. Check:

rdc gpus --json     # GPU capabilities
rdc counters --list # Available counters (empty if not supported)

Command Reference

For the complete list of all 66 commands with arguments, options, types, and defaults, see references/commands-quick-ref.md.

For extended debugging recipes with expected output shapes, see references/debugging-recipes.md.