debug-session

# Start Debug Session <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: CC-BY-4.0 --> Create a structured debugging session for an issue in the Dynamo ecosystem. ## Step 1: Get the Bug Report Ask the user how they want to provide the bug: **Option A: Linear ticket** - User provides ticket ID (e.g., "DYN-123") - Fetch via Linear MCP tools - Extract: title, description, reproduction steps **Option B: GitHub issue** - User provides issue URL - Fetch via `gh issue view <url>` - Extract: title, description, reproduction steps **Option C: Paste** - Ask user to paste the bug report directly - Parse out the key details ## Step 2: Discover Environment Gather environment information: !`nvidia-smi --query-gpu=name,count --format=csv,noheader 2>/dev/null || echo "No GPU detected"` !`uname -a` !`which python && python --version` This tells you: - GPU type and count (L40s, H100s, etc.) - OS/platform - Python environment **Note**: The user's `~/.claude/CLAUDE.md` may have more details about their dev environment (paths, aliases, preferences). Check there for additional context. ## Step 3: Create Worklog Create a worklog file to track the investigation: - Filename: `<issue-slug>.md` in current directory - Template: ```markdown # Debug: [Issue Title] **Date**: [today's date] **Source**: [Linear ticket / GitHub issue / user report] **Status**: investigating **Environment**: [GPU type/count from nvidia-smi] ## Problem [Description of the issue] ## Reproduction Steps 1. [Step to reproduce] 2. ... ## Expected vs Actual - **Expected**: - **Actual**: ## Investigation Log ### [timestamp] [Notes on what you tried/found] ## Root Cause [Fill in when found] ## Fix [Fill in when implemented] ``` ## Step 4: Set Up Testing ### Build Commands Rebuild Dynamo after making changes: ```bash cd lib/bindings/python && maturin develop --uv && cd ../../.. && uv pip install -e . ``` If a framework change is required (sglang, vllm, trtllm), check the user's `~/.claude/CLAUDE.md` for rebuild instructions specific to that framework. ### Running Examples Examples are located at the repository-relative path `examples/backends/`. Available backends: - `sglang/launch/` - SGLang backend examples - `vllm/launch/` - vLLM backend examples - `trtllm/launch/` - TensorRT-LLM backend examples Based on the bug report, determine which backend is relevant: - If unclear, **ask the user** which backend/example to run - Run the example in the background - Wait for model to be ready ### Verifying the Model is Up ```bash curl localhost:8000/v1/models ``` ### Testing with a Request ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "<model-name-from-above>", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50 }' ``` ## Step 5: Begin Investigation ### Dynamo Infrastructure Debugging **KV cache and routing issues:** - Check KV event logs in `lib/llm/src/block_manager/kv_consolidator/tracker.rs` - Look at block manager state and consolidation behavior - Inspect routing decisions in the KV-aware router **ZMQ / networking issues:** - Check ZMQ socket configuration and endpoint bindings - Look for connection timeouts or message drops - Verify nats/etcd connectivity for service discovery **Multi-node / disaggregated issues:** - Check prefill/decode worker assignment - Verify DGD (disaggregated) status reporting - Inspect inter-node communication via `nvidia-smi` on each node - Check NCCL and GPU direct RDMA status **Process inspection:** - `ps aux | grep dynamo` - check running processes - `nvidia-smi` - GPU utilization and memory - `ss -tlnp | grep 8000` - check port bindings - `journalctl -u dynamo` - systemd logs if applicable ### General Debugging Workflow 1. **Reproduce first** - verify you can trigger the bug before attempting fixes 2. **Document as you go** - update the worklog with findings 3. **Minimal changes** - fix the bug, do not refactor surrounding code 4. **Verify the fix** - confirm the reproduction case now passes Performance-critical code - avoid unnecessary abstractions or comments.