# Karpathy Neural Nets

> Use when building or debugging neural networks, deciding how to structure ML code for understanding, training a model that isn't converging, or approaching any complex technical system using Karpathy's "understand every line" pedagogy. Covers his recipe for training neural nets, Software 2.0 philosophy, and his build-from-scratch approach to deep understanding.

- Skill: `zap-coding-agent/karpathy-neural-nets` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zap-coding-agent/karpathy-neural-nets`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zap-coding-agent/karpathy-neural-nets/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: zap-coding-agent (https://skillmd.com/u/zap-coding-agent)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/zap-coding-agent/karpathy-neural-nets

---


# Andrej Karpathy — Neural Networks & Deep Understanding

## Core Philosophy

> "The most important skill in machine learning is the ability to recognize when something is wrong and diagnose why." — Karpathy

Karpathy's work (nanoGPT, micrograd, minBPE, his Stanford CS231n lectures, YouTube series) shares one unifying principle: **build it from scratch, then you truly understand it.** Not to be original, but because understanding every line is the only way to debug effectively, extend correctly, and teach clearly.

His approach is anti-abstraction at the learning phase: use PyTorch at the lowest level, avoid high-level frameworks when understanding is the goal. Use them only after you've built the thing yourself once.

## The Recipe for Training Neural Networks

From his widely-referenced 2019 essay — the checklist every practitioner should run before assuming the model is wrong:

### Phase 1: Become one with the data
```python
# Before writing a single line of model code:
# 1. Inspect raw samples — visualize, print, understand the distribution
# 2. Find duplicates, label errors, class imbalances
# 3. Look for patterns a human can use — if you can't, the model can't
# 4. Build a simple baseline (logistic regression, heuristic) FIRST
```

**Failure it prevents:** Models "failing" because the data has a bug, not the model.

### Phase 2: Set up the training/eval skeleton
- Fix the random seed for reproducibility.
- Start with a simplified version of the problem (fewer classes, smaller dataset).
- Add visualization: loss curves, sample predictions, gradient norms — *before* training a big run.
- Overfit a single batch first. If you can't fit 1 batch, something is fundamentally wrong.

```python
# The single-batch overfit test — non-negotiable
for _ in range(1000):
    loss = model(single_batch_X, single_batch_y)
    loss.backward()
    optimizer.step()
# loss should approach ~0. If it doesn't, the model/loss is broken.
```

### Phase 3: Find a good learning rate
Use a learning rate range test: increase LR exponentially across batches, plot LR vs loss. Pick the LR where loss decreases most steeply — not the minimum. Optimal LR is ~10× below where loss starts increasing.

### Phase 4: Train the full dataset
- Start with the simplest model that could work; complexity is a dial to turn up.
- Watch training vs validation loss diverge — that gap tells you where you are on the bias-variance tradeoff.
- Regularize based on what the gap says: more data, data augmentation, dropout, weight decay.

### Phase 5: Tune
Only now, hyperparameter search. By hand first (understand the direction), random search second, grid search almost never.

## Software 2.0 — the Paradigm Shift

From Karpathy's 2017 essay, which describes what he calls "the most important idea in ML":

> "In Software 1.0, we write code. In Software 2.0, we write training data and loss functions — the weights are the code."

The implications:
- **Neural networks are programs.** They have inputs, outputs, and bugs — they're just specified in weight space rather than source code.
- **Data is source code.** A mislabeled training example is a bug. Dataset curation is software engineering.
- **Debugging is different**: you can't read the "code" (the weights), but you can understand it through the data it learned from and the examples it fails on.
- **The architecture is the inductive bias.** Choosing a CNN over an MLP is choosing a prior: "spatial locality matters." Choosing a Transformer is choosing: "attention across the full sequence matters."

## Build From Scratch — the Karpathy Method

Every major Karpathy project demonstrates this: micrograd (autograd engine in 150 lines), nanoGPT (GPT-2 in 300 lines of PyTorch), minBPE (byte-pair encoding from scratch).

The method:
1. **Remove all abstractions.** Start with numpy arrays or the lowest-level primitive available.
2. **Implement the forward pass.** Verify it produces the right output on a known input.
3. **Implement the backward pass by hand.** Derive the gradients. Verify with `torch.autograd.gradcheck`.
4. **Add one abstraction at a time.** Each layer of the framework is understandable because you built the layer below it.
5. **Compare to the reference implementation.** Make sure your outputs match.

```python
# The gradient check — Karpathy's standard for "did I implement backprop correctly?"
from torch.autograd import gradcheck
input = torch.randn(3, 4, dtype=torch.double, requires_grad=True)
result = gradcheck(my_function, input, eps=1e-6, atol=1e-4)
print(result)  # True means your gradient is correct
```

## Debugging Neural Networks — Karpathy's Mental Model

> "Neural networks fail silently. The loss goes down, the model 'works', and the predictions are wrong."

Diagnostic questions in order:
1. **Is the loss plausible at initialization?** For cross-entropy with C classes, initial loss should be ≈ `log(C)`. If it's wildly off, your softmax or data loading is broken.
2. **Can the model overfit a small dataset?** If not, the model capacity or learning rate is wrong.
3. **Is the gradient flow healthy?** Watch gradient norms per layer. Dead layers (norm ≈ 0) or exploding gradients are structural failures.
4. **Is the validation loss actually tracking?** A validation loss that's *lower* than training loss means train/val split is leaking.
5. **Are the failure cases systematic?** Plot the worst predictions — they reveal what the model is missing, not just that it's missing something.

## The Pedagogical Principle

> "If you can't explain it simply, you don't understand it well enough."

Karpathy's lectures and repos are remarkable for their clarity. The principle he applies: **each abstraction earns its existence by reducing two things simultaneously: code and cognitive load.** If an abstraction reduces code but increases the amount you need to hold in your head to understand it, it's a net negative.

This transfers to software engineering generally: write for the reader who is you, six months from now, having forgotten everything.