Karpathy Neural Nets

Andrej Karpathy — Neural Networks & Deep Understanding

Core Philosophy

"The most important skill in machine learning is the ability to recognize when something is wrong and diagnose why." — Karpathy

Karpathy's work (nanoGPT, micrograd, minBPE, his Stanford CS231n lectures, YouTube series) shares one unifying principle: build it from scratch, then you truly understand it. Not to be original, but because understanding every line is the only way to debug effectively, extend correctly, and teach clearly.

His approach is anti-abstraction at the learning phase: use PyTorch at the lowest level, avoid high-level frameworks when understanding is the goal. Use them only after you've built the thing yourself once.

The Recipe for Training Neural Networks

From his widely-referenced 2019 essay — the checklist every practitioner should run before assuming the model is wrong:

Phase 1: Become one with the data

# Before writing a single line of model code:
# 1. Inspect raw samples — visualize, print, understand the distribution
# 2. Find duplicates, label errors, class imbalances
# 3. Look for patterns a human can use — if you can't, the model can't
# 4. Build a simple baseline (logistic regression, heuristic) FIRST

Failure it prevents: Models "failing" because the data has a bug, not the model.

Phase 2: Set up the training/eval skeleton

• Fix the random seed for reproducibility.
• Start with a simplified version of the problem (fewer classes, smaller dataset).
• Add visualization: loss curves, sample predictions, gradient norms — before training a big run.
• Overfit a single batch first. If you can't fit 1 batch, something is fundamentally wrong.

# The single-batch overfit test — non-negotiable
for _ in range(1000):
    loss = model(single_batch_X, single_batch_y)
    loss.backward()
    optimizer.step()
# loss should approach ~0. If it doesn't, the model/loss is broken.

Phase 3: Find a good learning rate

Use a learning rate range test: increase LR exponentially across batches, plot LR vs loss. Pick the LR where loss decreases most steeply — not the minimum. Optimal LR is ~10× below where loss starts increasing.

Phase 4: Train the full dataset

• Start with the simplest model that could work; complexity is a dial to turn up.
• Watch training vs validation loss diverge — that gap tells you where you are on the bias-variance tradeoff.
• Regularize based on what the gap says: more data, data augmentation, dropout, weight decay.

Phase 5: Tune

Only now, hyperparameter search. By hand first (understand the direction), random search second, grid search almost never.

Software 2.0 — the Paradigm Shift

From Karpathy's 2017 essay, which describes what he calls "the most important idea in ML":

"In Software 1.0, we write code. In Software 2.0, we write training data and loss functions — the weights are the code."

The implications:

• Neural networks are programs. They have inputs, outputs, and bugs — they're just specified in weight space rather than source code.
• Data is source code. A mislabeled training example is a bug. Dataset curation is software engineering.
• Debugging is different: you can't read the "code" (the weights), but you can understand it through the data it learned from and the examples it fails on.
• The architecture is the inductive bias. Choosing a CNN over an MLP is choosing a prior: "spatial locality matters." Choosing a Transformer is choosing: "attention across the full sequence matters."

Build From Scratch — the Karpathy Method

Every major Karpathy project demonstrates this: micrograd (autograd engine in 150 lines), nanoGPT (GPT-2 in 300 lines of PyTorch), minBPE (byte-pair encoding from scratch).

The method:

1. Remove all abstractions. Start with numpy arrays or the lowest-level primitive available.
2. Implement the forward pass. Verify it produces the right output on a known input.
3. Implement the backward pass by hand. Derive the gradients. Verify with torch.autograd.gradcheck.
4. Add one abstraction at a time. Each layer of the framework is understandable because you built the layer below it.
5. Compare to the reference implementation. Make sure your outputs match.

# The gradient check — Karpathy's standard for "did I implement backprop correctly?"
from torch.autograd import gradcheck
input = torch.randn(3, 4, dtype=torch.double, requires_grad=True)
result = gradcheck(my_function, input, eps=1e-6, atol=1e-4)
print(result)  # True means your gradient is correct

Debugging Neural Networks — Karpathy's Mental Model

"Neural networks fail silently. The loss goes down, the model 'works', and the predictions are wrong."

Diagnostic questions in order:

1. Is the loss plausible at initialization? For cross-entropy with C classes, initial loss should be ≈ log(C). If it's wildly off, your softmax or data loading is broken.
2. Can the model overfit a small dataset? If not, the model capacity or learning rate is wrong.
3. Is the gradient flow healthy? Watch gradient norms per layer. Dead layers (norm ≈ 0) or exploding gradients are structural failures.
4. Is the validation loss actually tracking? A validation loss that's lower than training loss means train/val split is leaking.
5. Are the failure cases systematic? Plot the worst predictions — they reveal what the model is missing, not just that it's missing something.

The Pedagogical Principle

"If you can't explain it simply, you don't understand it well enough."

Karpathy's lectures and repos are remarkable for their clarity. The principle he applies: each abstraction earns its existence by reducing two things simultaneously: code and cognitive load. If an abstraction reduces code but increases the amount you need to hold in your head to understand it, it's a net negative.

This transfers to software engineering generally: write for the reader who is you, six months from now, having forgotten everything.