Karpathy Verification Loop
Generation is cheap; discrimination — reading output and judging it — is the capability that stays scarce. Karpathy states the two halves in two separate passages of the same thread: he braces for a slopacolypse across digital media, and, under Atrophy, he notes that writing code and reading code are different capabilities in the brain (X status/2015883857489522876, 26 Jan 2026, agent-coding notes).
On the loop: "usually they are doing the generation, and we as humans are doing the verification. It is in our interest to make this loop go as fast as possible." (YC AI Startup School Software 3.0 talk, 17 Jun 2025; transcript at latent.space/p/s3).
Your job as reviewer is not syntax. Agent code now fails the way "a slightly sloppy, hasty junior dev" would (same Jan 2026 thread) — conceptually.
Boundary. Rule 4 of karpathy-coding-loop is the generator checking its own success criterion mid-loop. This skill is the reviewer deciding whether the artifact leaves the room.
Review procedure
- Read as visual red/green diffs, in small chunks — never raw dumps. If the diff is too big to audit in one sitting, the task was too big: split it and regenerate rather than skimming.
- Hunt conceptual errors, not typos.
- Wrong assumptions silently run with — the #1 failure mode.
- System-design and security flaws. Karpathy caught Claude matching a successful Stripe payment to user credits by email address, missing that the Stripe checkout email may differ from the Google signup email; line-correct, design-wrong, fixed by passing a persistent user id in the request metadata (karpathy.bearblog.dev/vibe-coding-menugen, 27 Apr 2025).
- Bloat: unneeded abstraction, defensive try/except for impossible cases, configurability nobody asked for.
- Collateral damage: comments or code changed outside the task's scope.
- Deprecated or hallucinated APIs — on MenuGen, Claude produced roughly 1000 lines against what looked like deprecated Clerk APIs (vibe-coding-menugen, 27 Apr 2025).
- Verify the dumb stuff regardless of apparent brilliance. Jagged intelligence: a model that just did something genius can still get 9.11 vs 9.9 wrong (X status/1816531576228053133, 25 Jul 2024, jagged-intelligence coinage). Independently recheck arithmetic, counts, and trivially-checkable claims. Route every calculation through code execution, never mental math from weights.
- Demo != done. "Demo is works.any(), product is works.all()" — slide text from the Software 3.0 talk as recorded by latent.space/p/s3 (17 Jun 2025), not retrievable spoken words. Each additional nine of reliability costs the same amount of work again — the march of nines, as reported from the Dwarkesh Podcast interview (17 Oct 2025). And feeling 80% done usually means closer to 20%: on MenuGen the remaining bulk was deployment, auth, and browser-based config across four vendor consoles, not code (vibe-coding-menugen, 27 Apr 2025).
- High-stakes answer -> LLM council. Put the same question to a handful of frontier models (his council ships with four: gpt-5.1, gemini-3-pro-preview, claude-sonnet-4.5, grok-4), have them cross-review and rank each other with identities hidden, then apply your own qualitative read as the final ranking. As reported of his announcement, the council ranked GPT-5.1 top while he preferred Gemini 3's answer (X status/1992381094667411768, 22 Nov 2025, llm-council announcement).
LLM-as-judge
- Demand an exact machine-parseable output format and tell the judge why: "I will be parsing it programmatically" (karpathy.bearblog.dev/auto-grade-hn, 10 Dec 2025).
- Anonymize identities in the judged inputs. The llm-council README gives this as the design rationale — identities are hidden so a model cannot play favorites — not as a bias he measured.
- Spot-check verdicts by hand and iterate the rubric on the errors you find
(house rule).
Excuse -> Reality
| Excuse |
Reality |
| "The tests pass" |
Tests can encode the same wrong assumption the code does. Check the assumption, not the green. |
| "The diff is long but it's all boilerplate" |
Then it is cheap to split. Long-and-skimmed is how design flaws ship. |
| "It worked when I ran it" |
That is works.any(). Name the failure cases you did not run. |
| "The agent said it verified" |
Self-report is not evidence. Run the command yourself. |
| "I read the summary of the changes" |
The summary was generated by the thing under review. |
| "The math looks right" |
Route it through code. Jagged intelligence. |
Red flags
- Accepting a diff you scrolled past rather than read, outside declared throwaway mode.
- "The demo worked" as the completion criterion.
- Trusting a model's arithmetic or self-reported success without an independent check.
- Reviewing only what changed, not what the change implies for the system design.
If the verdict is environmental
A hallucinated API, a regression to conventions this repo does not use, or the third repeat of the same mistake is a context problem, not a review problem: go to karpathy-context-engineering rather than re-prompting.
Sources
Karpathy: YC AI Startup School Software 3.0 talk, 17 Jun 2025 (transcript, slide text and annotations at latent.space/p/s3) — generation/verification loop, works.any/works.all. karpathy.bearblog.dev/vibe-coding-menugen, 27 Apr 2025 — Stripe/email design flaw, deprecated Clerk APIs, 80%/20%. karpathy.bearblog.dev/auto-grade-hn, 10 Dec 2025 — parse-it-programmatically prompt. X status/2015883857489522876 (26 Jan 2026, agent-coding notes) — slopacolypse, generation vs discrimination, hasty-junior-dev failure mode. X status/1816531576228053133 (25 Jul 2024, jagged-intelligence coinage). X status/1992381094667411768 (22 Nov 2025, llm-council announcement) — the four council models and the GPT-5.1-vs-Gemini-3 disagreement, as reported. github.com/karpathy/llm-council README — the anonymization design only; it does not report the disagreement. Dwarkesh Podcast, 17 Oct 2025 — march of nines, via secondary reporting; the phrase was not retrievable in the published transcript.
MIT notice for this skill set (upstream: multica-ai/andrej-karpathy-skills, formerly forrestchang/andrej-karpathy-skills): see karpathy-coding-loop/references/examples.md.
1---2name: karpathy-verification3description: Use when judging whether output is correct or finished: reviewing an AI-written diff before accepting, merging, or shipping; deciding whether a working demo means done; checking a model's numbers, counts, or self-reported success; or setting up an LLM as a judge, grader, or council.4---56# Karpathy Verification Loop78Generation is cheap; discrimination — reading output and judging it — is the capability that stays scarce. Karpathy states the two halves in two separate passages of the same thread: he braces for a slopacolypse across digital media, and, under Atrophy, he notes that writing code and reading code are different capabilities in the brain (X status/2015883857489522876, 26 Jan 2026, agent-coding notes).910On the loop: "usually they are doing the generation, and we as humans are doing the verification. It is in our interest to make this loop go as fast as possible." (YC AI Startup School Software 3.0 talk, 17 Jun 2025; transcript at latent.space/p/s3).1112Your job as reviewer is not syntax. Agent code now fails the way "a slightly sloppy, hasty junior dev" would (same Jan 2026 thread) — conceptually.1314**Boundary.** Rule 4 of `karpathy-coding-loop` is the generator checking its own success criterion mid-loop. This skill is the reviewer deciding whether the artifact leaves the room.1516## Review procedure17181. **Read as visual red/green diffs, in small chunks — never raw dumps.** If the diff is too big to audit in one sitting, the task was too big: split it and regenerate rather than skimming.192. **Hunt conceptual errors, not typos.**20 - Wrong assumptions silently run with — the #1 failure mode.21 - System-design and security flaws. Karpathy caught Claude matching a successful Stripe payment to user credits by email address, missing that the Stripe checkout email may differ from the Google signup email; line-correct, design-wrong, fixed by passing a persistent user id in the request metadata (karpathy.bearblog.dev/vibe-coding-menugen, 27 Apr 2025).22 - Bloat: unneeded abstraction, defensive try/except for impossible cases, configurability nobody asked for.23 - Collateral damage: comments or code changed outside the task's scope.24 - Deprecated or hallucinated APIs — on MenuGen, Claude produced roughly 1000 lines against what looked like deprecated Clerk APIs (vibe-coding-menugen, 27 Apr 2025).253. **Verify the dumb stuff regardless of apparent brilliance.** Jagged intelligence: a model that just did something genius can still get 9.11 vs 9.9 wrong (X status/1816531576228053133, 25 Jul 2024, jagged-intelligence coinage). Independently recheck arithmetic, counts, and trivially-checkable claims. Route every calculation through code execution, never mental math from weights.264. **Demo != done.** "Demo is works.any(), product is works.all()" — slide text from the Software 3.0 talk as recorded by latent.space/p/s3 (17 Jun 2025), not retrievable spoken words. Each additional nine of reliability costs the same amount of work again — the march of nines, as reported from the Dwarkesh Podcast interview (17 Oct 2025). And feeling 80% done usually means closer to 20%: on MenuGen the remaining bulk was deployment, auth, and browser-based config across four vendor consoles, not code (vibe-coding-menugen, 27 Apr 2025).275. **High-stakes answer -> LLM council.** Put the same question to a handful of frontier models (his council ships with four: gpt-5.1, gemini-3-pro-preview, claude-sonnet-4.5, grok-4), have them cross-review and rank each other with identities hidden, then apply your own qualitative read as the final ranking. As reported of his announcement, the council ranked GPT-5.1 top while he preferred Gemini 3's answer (X status/1992381094667411768, 22 Nov 2025, llm-council announcement).2829## LLM-as-judge3031- Demand an exact machine-parseable output format and tell the judge why: "I will be parsing it programmatically" (karpathy.bearblog.dev/auto-grade-hn, 10 Dec 2025).32- Anonymize identities in the judged inputs. The llm-council README gives this as the design rationale — identities are hidden so a model cannot play favorites — not as a bias he measured.33- Spot-check verdicts by hand and iterate the rubric on the errors you find `(house rule)`.3435## Excuse -> Reality3637| Excuse | Reality |38|---|---|39| "The tests pass" | Tests can encode the same wrong assumption the code does. Check the assumption, not the green. |40| "The diff is long but it's all boilerplate" | Then it is cheap to split. Long-and-skimmed is how design flaws ship. |41| "It worked when I ran it" | That is works.any(). Name the failure cases you did not run. |42| "The agent said it verified" | Self-report is not evidence. Run the command yourself. |43| "I read the summary of the changes" | The summary was generated by the thing under review. |44| "The math looks right" | Route it through code. Jagged intelligence. |4546## Red flags4748- Accepting a diff you scrolled past rather than read, outside declared throwaway mode.49- "The demo worked" as the completion criterion.50- Trusting a model's arithmetic or self-reported success without an independent check.51- Reviewing only what changed, not what the change implies for the system design.5253## If the verdict is environmental5455A hallucinated API, a regression to conventions this repo does not use, or the third repeat of the same mistake is a context problem, not a review problem: go to `karpathy-context-engineering` rather than re-prompting.5657## Sources5859Karpathy: YC AI Startup School Software 3.0 talk, 17 Jun 2025 (transcript, slide text and annotations at latent.space/p/s3) — generation/verification loop, works.any/works.all. karpathy.bearblog.dev/vibe-coding-menugen, 27 Apr 2025 — Stripe/email design flaw, deprecated Clerk APIs, 80%/20%. karpathy.bearblog.dev/auto-grade-hn, 10 Dec 2025 — parse-it-programmatically prompt. X status/2015883857489522876 (26 Jan 2026, agent-coding notes) — slopacolypse, generation vs discrimination, hasty-junior-dev failure mode. X status/1816531576228053133 (25 Jul 2024, jagged-intelligence coinage). X status/1992381094667411768 (22 Nov 2025, llm-council announcement) — the four council models and the GPT-5.1-vs-Gemini-3 disagreement, as reported. github.com/karpathy/llm-council README — the anonymization design only; it does not report the disagreement. Dwarkesh Podcast, 17 Oct 2025 — march of nines, via secondary reporting; the phrase was not retrievable in the published transcript.6061MIT notice for this skill set (upstream: `multica-ai/andrej-karpathy-skills`, formerly `forrestchang/andrej-karpathy-skills`): see `karpathy-coding-loop/references/examples.md`.
yshms