How do I install the Dgx Diagnose skill?

Run npx skillmds@latest add nvidia/dgx-diagnose in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Dgx Diagnose safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Dgx Diagnose?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Dgx Diagnose free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Dgx Diagnose?

NVIDIA (@nvidia) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Dgx Diagnose by nvidia — Agent Skill

Name: Dgx Diagnose
Author: NVIDIA

Dgx Diagnose

Diagnose common DGX Station GB300 issues — CUDA crashes, wrong-GPU targeting, vLLM/SGLang container bugs, MIG state problems, NVLink/Fabric Manager errors, X/Vulkan failures, HuggingFace auth, and port conflicts. Use when the user reports a GPU error, inference server crash, MIG problem, or any unexplained DGX Station failure.

NVIDIA d466214 3.3 KB Updated Sep 17, 2026 2.2k repo stars

File contents

DGX Station Diagnostics

Diagnose common DGX Station issues. Run through the checks below to identify the problem.

Step 1. Gather system state

Run these commands and analyze the output:

# GPU status
nvidia-smi

# GPU device list with indices
nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader

# Driver version
nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1

# MIG state
nvidia-smi -i 1 -q 2>/dev/null | grep -i "MIG Mode" || echo "Could not query MIG on device 1"

# Fabric Manager
systemctl is-active nvidia-fabricmanager

# GPU processes
sudo fuser -v /dev/nvidia* 2>/dev/null || echo "No GPU processes found"

# Docker containers using GPUs
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" 2>/dev/null

Step 2. Match symptoms to known issues

Based on the gathered state and the user's reported problem, check for these known issues:

CUDA crashes with `--gpus all`

Cause: Mixed coherency — GB300 (ATS) and RTX PRO (non-ATS) cannot share a CUDA context. Fix: Use --gpus '"device=N"' targeting only the GB300.

Model running on wrong GPU (RTX PRO instead of GB300)

Check: The device index in the docker command vs actual GPU indices. Fix: Verify with nvidia-smi --query-gpu=index,name --format=csv,noheader and correct the --gpus flag.

vLLM crash / FlashInfer buffer overflow

Check: Container version — docker inspect vllm-server | grep Image Fix: Use nvcr.io/nvidia/vllm:26.01-py3. Version 25.10 has a known FlashInfer bug on DGX Station.

SGLang CUDA errors

Check: Container tag — must be cu130 for Blackwell SM103. Fix: Use lmsysorg/sglang:latest-cu130.

CUDA OOM despite 279 GB HBM

Check: --max-model-len / --context-length and memory utilization settings. Fix: Reduce context length or lower --gpu-memory-utilization / --mem-fraction-static.

`nvidia-smi -mig 1` returns "In use by another client"

Check: sudo fuser -v /dev/nvidia* — GPU processes must be stopped first. Fix: Stop all GPU workloads, then retry.

NVLink errors after disabling MIG

Check: systemctl is-active nvidia-fabricmanager Fix: sudo systemctl start nvidia-fabricmanager

X server crash after nvidia-xconfig -a

Fix: sudo cp /etc/X11/xorg.conf.nvidia-xconfig-original /etc/X11/xorg.conf

Vulkan VK_ERROR_INITIALIZATION_FAILED

Cause: CUDA initialized before Vulkan, binding to GB300. Fix: Run CUDA and Vulkan workloads in separate processes. For Vulkan apps: __GL_DeviceModalityPreference=2 ./your_app

HuggingFace 401 / token errors

Fix: Pass token inline: -e HF_TOKEN="hf_...". Don't rely on shell export for background Docker tasks.

Port already in use

Check: lsof -i :<PORT> Fix: Stop the conflicting process or use a different host port: -p 8001:8000.

Step 3. Report findings

Tell the user:

What the issue is
Why it happens (root cause)
The specific command to fix it
How to verify the fix worked

---name: dgx-diagnosedescription: Diagnose common DGX Station GB300 issues — CUDA crashes, wrong-GPU targeting, vLLM/SGLang container bugs, MIG state problems, NVLink/Fabric Manager errors, X/Vulkan failures, HuggingFace auth, and port conflicts. Use when the user reports a GPU error, inference server crash, MIG problem, or any unexplained DGX Station failure.---# DGX Station DiagnosticsDiagnose common DGX Station issues. Run through the checks below to identify the problem.## Step 1. Gather system stateRun these commands and analyze the output:```bash# GPU statusnvidia-smi# GPU device list with indicesnvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader# Driver versionnvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1# MIG statenvidia-smi -i 1 -q 2>/dev/null | grep -i "MIG Mode" || echo "Could not query MIG on device 1"# Fabric Managersystemctl is-active nvidia-fabricmanager# GPU processessudo fuser -v /dev/nvidia* 2>/dev/null || echo "No GPU processes found"# Docker containers using GPUsdocker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" 2>/dev/null```## Step 2. Match symptoms to known issuesBased on the gathered state and the user's reported problem, check for these known issues:### CUDA crashes with `--gpus all`**Cause:** Mixed coherency — GB300 (ATS) and RTX PRO (non-ATS) cannot share a CUDA context.**Fix:** Use `--gpus '"device=N"'` targeting only the GB300.### Model running on wrong GPU (RTX PRO instead of GB300)**Check:** The device index in the docker command vs actual GPU indices.**Fix:** Verify with `nvidia-smi --query-gpu=index,name --format=csv,noheader` and correct the `--gpus` flag.### vLLM crash / FlashInfer buffer overflow**Check:** Container version — `docker inspect vllm-server | grep Image`**Fix:** Use `nvcr.io/nvidia/vllm:26.01-py3`. Version 25.10 has a known FlashInfer bug on DGX Station.### SGLang CUDA errors**Check:** Container tag — must be `cu130` for Blackwell SM103.**Fix:** Use `lmsysorg/sglang:latest-cu130`.### CUDA OOM despite 279 GB HBM**Check:** `--max-model-len` / `--context-length` and memory utilization settings.**Fix:** Reduce context length or lower `--gpu-memory-utilization` / `--mem-fraction-static`.### `nvidia-smi -mig 1` returns "In use by another client"**Check:** `sudo fuser -v /dev/nvidia*` — GPU processes must be stopped first.**Fix:** Stop all GPU workloads, then retry.### NVLink errors after disabling MIG**Check:** `systemctl is-active nvidia-fabricmanager`**Fix:** `sudo systemctl start nvidia-fabricmanager`### X server crash after nvidia-xconfig -a**Fix:** `sudo cp /etc/X11/xorg.conf.nvidia-xconfig-original /etc/X11/xorg.conf`### Vulkan VK_ERROR_INITIALIZATION_FAILED**Cause:** CUDA initialized before Vulkan, binding to GB300.**Fix:** Run CUDA and Vulkan workloads in separate processes. For Vulkan apps: `__GL_DeviceModalityPreference=2 ./your_app`### HuggingFace 401 / token errors**Fix:** Pass token inline: `-e HF_TOKEN="hf_..."`. Don't rely on shell export for background Docker tasks.### Port already in use**Check:** `lsof -i :<PORT>`**Fix:** Stop the conflicting process or use a different host port: `-p 8001:8000`.## Step 3. Report findingsTell the user:1. What the issue is2. Why it happens (root cause)3. The specific command to fix it4. How to verify the fix worked

Frequently asked questions