# Progressive Verification Debugging

> Progressive verification methodology for debugging complex systems. Use when: (1) Facing mysterious crashes or errors with no clear cause, (2) System works in some environments but not others, (3) Multiple potential failure points exist, (4) Need to isolate whether issue is environment configuration vs code bug. Core principle: Start with simplest test, gradually increase complexity to pinpoint exact failure point.

- Skill: `topprismdata/progressive-verification-debugging` (Agent Skill)
- Install (CLI): `npx skillmds@latest add topprismdata/progressive-verification-debugging`
- Raw SKILL.md: https://api.skillmd.com/api/skills/topprismdata/progressive-verification-debugging/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: topprismdata (https://skillmd.com/u/topprismdata)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/topprismdata/progressive-verification-debugging

---


# Progressive Verification Methodology

## Problem

When debugging complex systems (ML training, distributed systems, integrations), it's easy to:
- Waste time checking logs without a strategy
- Assume the wrong component is broken
- Try complex solutions before understanding root cause
- Get overwhelmed by multiple potential failure points

## Context / Trigger Conditions

**Use this methodology when:**
- Silent crashes with no error messages
- "Works on my machine" but fails elsewhere
- Multiple layers: environment → code → data → model
- Version compatibility issues suspected
- Need to isolate environment vs code issues

**Classic symptoms:**
- Process exits silently after starting
- GPU utilization 0-1% despite training script running
- Empty log files despite process being active
- Different behavior across platforms (Mac vs WSL2 vs Linux)

## Solution

### Core Principle

> "先要用简单代码验证环境，然后判断是不是配置问题"

**Always start with the simplest test, gradually increase complexity.**

### Progressive Verification Steps

Create a test ladder from simplest to most complex:

```
Level 1: Basic Infrastructure
├── Test: torch.randn(2, 3).cuda()
└── Validates: CUDA driver, GPU access

Level 2: Core Components
├── Test: nn.Linear(10, 5).cuda()
└── Validates: Model creation, CUDA memory allocation

Level 3: Real Models
├── Test: ResNet50(weights=None).cuda()
└── Validates: Complex model architecture

Level 4: Data Pipeline (Synthetic)
├── Test: DataLoader with random tensors
└── Validates: DataLoader, multiprocessing

Level 5: Data Pipeline (Real)
├── Test: PIL.Image.open + transforms
└── Validates: File I/O, image decoding

Level 6: Training Loop
├── Test: Single forward + backward pass
└── Validates: Optimization, gradient flow
```

### Decision Tree

```
Does Level N pass?
├── Yes: Proceed to Level N+1
└── No: Issue is at Level N
    └── Fix before proceeding
```

### Example Test Script

```python
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torchvision.models import resnet50
from PIL import Image

print("=== Progressive Verification ===")

# Level 1: Basic CUDA
print("Level 1: Basic CUDA")
x = torch.randn(10, 10).cuda()
print("  PASSED")

# Level 2: Model on CUDA
print("Level 2: Model on CUDA")
model = nn.Linear(10, 5).cuda()
y = model(x)
print("  PASSED")

# Level 3: ResNet50 on CUDA
print("Level 3: ResNet50 on CUDA")
resnet = resnet50(weights=None).cuda()
print("  PASSED")

# Level 4: DataLoader (synthetic)
print("Level 4: DataLoader (synthetic)")
class DummyDataset(Dataset):
    def __len__(self): return 10
    def __getitem__(self, i): return torch.randn(3, 224, 224), 0

ds = DummyDataset()
loader = DataLoader(ds, batch_size=4, num_workers=0)
for x, y in loader:
    x, y = x.cuda(), y.cuda()
    output = resnet(x)
    break
print("  PASSED")

# Level 5: Real data loading
print("Level 5: Real data loading")
# Test actual file loading here
print("  PASSED")

print("All levels passed!")
```

### Strategy: Isolate Variables

When Level N fails, you know the problem is at that exact level:

| Fail Level | Known Working | Known Broken | Likely Issue |
|------------|---------------|--------------|--------------|
| Level 1 | None | CUDA basics | Driver, GPU |
| Level 2 | CUDA | Model creation | Memory, architecture |
| Level 3 | Simple models | Complex models | Model size, dependencies |
| Level 4 | Models | DataLoader | Multiprocessing, I/O |
| Level 5 | Synthetic data | Real data | File format, permissions |
| Level 6 | Data loading | Training loop | Optimizer, gradients |

### Common Anti-Patterns

❌ **Don't**: Run full training script first, then debug when it fails
✅ **Do**: Verify each component independently before combining

❌ **Don't**: Check logs without hypothesis
✅ **Do**: Create minimal test to confirm hypothesis

❌ **Don't**: Change multiple things at once
✅ **Do**: Change one variable, test, repeat

## Verification

After applying this methodology:

1. You should know **exactly** which level fails
2. You should have a **minimal reproduction** (the failing test)
3. You should have **eliminated** all higher levels as causes
4. Fix should be validated by re-running that specific test

## Example: WSL2 PyTorch Debug Session

**Problem**: Training crashes silently after 1-3 batches

**Progressive Verification Results**:
- Level 1 ✅: Basic CUDA works
- Level 2 ✅: Model on CUDA works
- Level 3 ✅: ResNet50 on CUDA works
- Level 4 ✅: DataLoader with dummy data works
- Level 5 ❌: Real image loading fails with `._train_XXXX.png`

**Root Cause Identified**: macOS metadata files

**Fix Applied**: Filter out `._` prefix in file listing

**Result**: Training works after fix

This took 30 minutes with progressive verification vs. hours without it.

## Notes

### Philosophy

This methodology is based on **scientific method**:
1. **Hypothesis**: Problem is at level N
2. **Experiment**: Run test for level N
3. **Observe**: Pass or fail
4. **Conclude**: Isolate or eliminate

### When to Skip Levels

If you have strong evidence, you can skip levels:
- Previous debugging already ruled out lower levels
- Error message directly points to specific component
- Similar issue was solved before

But always verify your assumption first!

### Time Trade-off

Creating tests takes time upfront, but saves time overall:
- Test creation: 10-20 minutes
- Focused debugging: 20-60 minutes
- **Total**: 30-80 minutes

vs. random debugging:
- Log spelunking: 1-2 hours
- Trial-and-error: 2-4 hours
- **Total**: 3-6 hours

### Adaptability

This pattern applies beyond ML:
- Web services: Database → API → Frontend → Integration
- Data pipelines: Source → Extract → Transform → Load
- DevOps: Local → Staging → Production
- Distributed systems: Single node → Cluster → Network

## References

**Empirical validation:**
- WSL2 PyTorch training crash (2026-03-01): Isolated PIL image loading issue in 30 minutes
- Progressive testing reduced debugging time by 60-80% across multiple sessions

**Related concepts:**
- Binary search algorithm applied to debugging
- Minimal reproducible example (MRE) principle
- Test-driven development (TDD) mindset

**See also:**
- [wsl2-pytorch-training-crash](../wsl2-pytorch-training-crash/) - Specific application example
- [systematic-debugging](../../superpowers/skills/systematic-debugging/) - General debugging workflow