# Debug Session

> Sets up a structured debugging session for a Dynamo bug — pull the report from a Linear ticket, GitHub issue, or pasted text, capture the environment, create a persistent worklog markdown file, and walk through rebuild, reproduction, and investigation steps. Use when starting to investigate a reported Dynamo bug or regression and the investigation should be tracked in a worklog.

- Skill: `ai-dynamo-dynamo/debug-session` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ai-dynamo-dynamo/debug-session`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ai-dynamo-dynamo/debug-session/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- License: Apache-2.0
- Author: ai-dynamo (https://skillmd.com/u/ai-dynamo-dynamo)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/ai-dynamo-dynamo/debug-session

---


# Start Debug Session

<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: CC-BY-4.0
-->

Create a structured debugging session for an issue in the Dynamo ecosystem.

## Step 1: Get the Bug Report

Ask the user how they want to provide the bug:

**Option A: Linear ticket**
- User provides ticket ID (e.g., "DYN-123")
- Fetch via Linear MCP tools
- Extract: title, description, reproduction steps

**Option B: GitHub issue**
- User provides issue URL
- Fetch via `gh issue view <url>`
- Extract: title, description, reproduction steps

**Option C: Paste**
- Ask user to paste the bug report directly
- Parse out the key details

## Step 2: Discover Environment

Gather environment information:

!`nvidia-smi --query-gpu=name,count --format=csv,noheader 2>/dev/null || echo "No GPU detected"`

!`uname -a`

!`which python && python --version`

This tells you:
- GPU type and count (L40s, H100s, etc.)
- OS/platform
- Python environment

**Note**: The user's `~/.claude/CLAUDE.md` may have more details about their dev environment (paths, aliases, preferences). Check there for additional context.

## Step 3: Create Worklog

Create a worklog file to track the investigation:

- Filename: `<issue-slug>.md` in current directory
- Template:

```markdown
# Debug: [Issue Title]

**Date**: [today's date]
**Source**: [Linear ticket / GitHub issue / user report]
**Status**: investigating
**Environment**: [GPU type/count from nvidia-smi]

## Problem
[Description of the issue]

## Reproduction Steps
1. [Step to reproduce]
2. ...

## Expected vs Actual
- **Expected**:
- **Actual**:

## Investigation Log

### [timestamp]
[Notes on what you tried/found]

## Root Cause
[Fill in when found]

## Fix
[Fill in when implemented]
```

## Step 4: Set Up Testing

### Build Commands

Rebuild Dynamo after making changes:
```bash
cd lib/bindings/python && maturin develop --uv && cd ../../.. && uv pip install -e .
```

If a framework change is required (sglang, vllm, trtllm), check the user's `~/.claude/CLAUDE.md` for rebuild instructions specific to that framework.

### Running Examples

Examples are located at the repository-relative path `examples/backends/`.

Available backends:
- `sglang/launch/` - SGLang backend examples
- `vllm/launch/` - vLLM backend examples
- `trtllm/launch/` - TensorRT-LLM backend examples

Based on the bug report, determine which backend is relevant:
- If unclear, **ask the user** which backend/example to run
- Run the example in the background
- Wait for model to be ready

### Verifying the Model is Up

```bash
curl localhost:8000/v1/models
```

### Testing with a Request

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name-from-above>",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 50
  }'
```

## Step 5: Begin Investigation

### Dynamo Infrastructure Debugging

**KV cache and routing issues:**
- Check KV event logs in `lib/llm/src/block_manager/kv_consolidator/tracker.rs`
- Look at block manager state and consolidation behavior
- Inspect routing decisions in the KV-aware router

**ZMQ / networking issues:**
- Check ZMQ socket configuration and endpoint bindings
- Look for connection timeouts or message drops
- Verify nats/etcd connectivity for service discovery

**Multi-node / disaggregated issues:**
- Check prefill/decode worker assignment
- Verify DGD (disaggregated) status reporting
- Inspect inter-node communication via `nvidia-smi` on each node
- Check NCCL and GPU direct RDMA status

**Process inspection:**
- `ps aux | grep dynamo` - check running processes
- `nvidia-smi` - GPU utilization and memory
- `ss -tlnp | grep 8000` - check port bindings
- `journalctl -u dynamo` - systemd logs if applicable

### General Debugging Workflow

1. **Reproduce first** - verify you can trigger the bug before attempting fixes
2. **Document as you go** - update the worklog with findings
3. **Minimal changes** - fix the bug, do not refactor surrounding code
4. **Verify the fix** - confirm the reproduction case now passes

Performance-critical code - avoid unnecessary abstractions or comments.