# Ml Debug

> Debug ML issues systematically -- tensor shapes, NaN propagation, gradient flow, device mismatches, dtype errors. Use when encountering training failures, unexpected outputs, or numerical issues.

- Skill: `giacomogaglione/ml-debug` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add giacomogaglione/ml-debug`
- Raw SKILL.md: https://api.skillmd.com/api/skills/giacomogaglione/ml-debug/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: giacomogaglione (https://skillmd.com/u/giacomogaglione)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/giacomogaglione/ml-debug

---


# ML Debugging Skill

When debugging an ML issue, work through these checks in order. Stop at the first check that reveals the problem.

## 1. Shape Verification

Trace tensor shapes through the computation:

- Print shapes at every function boundary: `print(f"input: {x.shape}, output: {y.shape}")`
- Verify batch dimension is preserved through all operations
- Check that reshape/view operations don't silently permute data
- Verify attention mask shapes match query/key dimensions

## 2. Dtype and Device Checks

Look for silent type/device mismatches:

- Print `tensor.dtype` and `tensor.device` at suspicious points
- Check for float32/float64 mixing (common in loss computation)
- Verify all tensors in an operation are on the same device
- Check for integer overflow in index operations

## 3. NaN/Inf Propagation

Trace where NaN or Inf first appears:

- Insert `assert not torch.isnan(x).any(), f"NaN at {name}"` after each operation
- Enable anomaly detection: `torch.autograd.set_detect_anomaly(True)`
- Check for division by zero in normalization layers
- Check for log(0) or log(negative) in loss functions
- Check for extremely large values before softmax

## 4. Gradient Flow

Diagnose vanishing or exploding gradients:

- Print gradient norms: `torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)`
- Check if any parameter has `requires_grad=False` unintentionally
- Check for dead ReLU units (all-zero activations)
- Verify loss is connected to all trainable parameters

## 5. Data Pipeline

Check if the issue is in the data, not the model:

- Verify labels match inputs (visualize a few samples)
- Check for data leakage between train/val/test
- Verify normalization statistics (mean, std) are computed on training set only
- Check class imbalance
- Verify data augmentation isn't corrupting labels

## 6. Common Library Gotchas

### PyTorch
- `model.eval()` vs `model.train()` -- affects dropout and batchnorm
- `torch.no_grad()` must wrap inference code
- `loss.backward()` accumulates gradients -- call `optimizer.zero_grad()` first
- `DataLoader` with `num_workers > 0` can hide errors in worker processes

### pandas
- Index alignment: `df1 + df2` aligns on index, not position
- `SettingWithCopyWarning` means you're modifying a view, not a copy
- `groupby().apply()` can call the function twice on the first group

### numpy
- Broadcasting: `(3,1) * (1,4)` gives `(3,4)` -- verify this is intended
- `np.array` copies by default, `np.asarray` does not
- Integer arrays silently overflow