# Sherlock > Sherlock — root-cause by elimination - Skill: `drubinstein/sherlock` (Agent Skill, multi-file: 5 files) - Install (CLI): `npx skillmds@latest add drubinstein/sherlock` - Raw SKILL.md: https://api.skillmd.com/api/skills/drubinstein/sherlock/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: drubinstein (https://skillmd.com/u/drubinstein) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/drubinstein/sherlock --- # Sherlock — root-cause by elimination > *"When you have eliminated the impossible, whatever remains, however improbable, > must be the truth."* — Sherlock Holmes, *The Sign of the Four* Sherlock is the **differential-diagnosis discipline** for bugs. Instead of chasing your favorite theory, you build the **complete list of possible causes**, then **eliminate each one with decisive evidence** — never with a hunch or a reputation — until a single explanation survives. Whatever survives is the truth, however much it "shouldn't" be possible. The whole method turns on one asymmetry: **you cannot confirm your way to a root cause, but you can eliminate your way there.** Confirmation flatters the theory you already hold; elimination is forced to consider the ones you don't. Six moves, each a guard against the way bug-hunts actually fail: - **The complete suspect list.** Enumerate *all* candidate causes up front — including the ones you "know" can't be it. An **incomplete list** is the #1 reason a bug "can't be found": the true cause was never written down. - **Eliminate by proof, not by reputation.** A suspect leaves the board only when a test makes it *impossible*. "The stdlib is well-tested," "we decommissioned that," "the compiler doesn't have bugs" are **not evidence** — they're the exact assumptions the real bug hides behind. - **Falsify, don't confirm.** Design the test that would **kill** your leading theory, not the one that flatters it. *Twist theories to fit facts; never twist facts to fit theories.* - **The dog that didn't bark.** The **absence** of an expected signal — a missing log line, an error that never fired, a retry that never happened — is evidence. *Observe*, don't merely *see*. - **Trust the improbable survivor.** When one cause remains, accept it even though it "can't" happen, then **confirm** it. The bug that can't happen *is* happening. - **Zero survivors ⇒ your reasoning failed, not the universe.** If you've eliminated everything, you either **over-eliminated** (killed a suspect on assumption, not proof — re-test it) or **under-enumerated** (the cause is one you dismissed as impossible — widen the list). This branch is the maxim itself. ## When to use Reach for Sherlock when the obvious explanation is gone and you're **choosing among many candidate causes**: - a "this is impossible / can't be happening" bug — a heisenbug, or one that vanishes under the debugger / with logging added - intermittent, flaky, or nondeterministic failures with several plausible causes - a regression where many things could be the culprit (which commit? which layer? which dependency bump?) - works-on-my-machine / fails-in-CI / fails-only-in-prod environment bugs - the moment you catch yourself saying **"I've checked everything and nothing explains it"** — that sentence is the signal to run this method - two or three theories left and you're tempted to just fix the likely-looking one ## When NOT to use - The cause is **obvious** — a clear stack trace pointing at one line. Just fix it. (But if the "obvious" fix doesn't make the symptom vanish, the cause wasn't obvious — open a board.) - You have **no facts yet** — nothing reproduced, no logs, no trace. *"It is a capital mistake to theorise before one has data."* First make it observable and reproduce it (instrument-first — see `andrej` / superpowers:systematic-debugging), THEN enumerate suspects. (But a *reported* guarantee — "we have a UNIQUE constraint," "the logs show one request," "every dashboard is green" — is a claim to **verify**, not a fact: green dashboards are not "no facts," and an unverified guarantee goes on the board as a *suspect* until you observe it directly.) - A pure **performance** "why is it slow" with no discrete wrong behavior — that's a resource whodunit; run `brendan`'s USE sweep (it *is* elimination, specialized to resources). Sherlock is for **correctness** bugs (a slow path that *also* returns wrong data is a correctness bug — use Sherlock). - A feature/design task with no defect to explain. ## The elimination loop 1. **OBSERVE — collect the facts, no theory yet.** Pin the *exact* symptom and gather hard evidence: a reproduction, logs, traces, the failing diff, inputs. Write down the **negative facts** too — what should have happened and didn't (the dog that didn't bark). Resist theorizing until the facts are on the table. 2. **ENUMERATE — the complete suspect list.** List *every* cause that could produce these exact facts. Probability is irrelevant here; **deliberately include the improbable and the "impossible."** Sweep the layers so the list is complete: your code · your config · the inputs/data · build & toolchain · dependencies · runtime/OS · hardware · environment/network · concurrency & timing · state & caching · **the observer/test harness itself**. 3. **ELIMINATE — by decisive, falsifying test.** For each suspect, find the *cheapest* observation that can prove it **impossible**, and run it. Prefer tests that **bisect** the list (kill half at once). Record *how* each suspect was eliminated — the evidence, not the argument. **A suspect you cannot yet disprove stays on the board.** Never strike one off because "it can't be that." 4. **CONVERGE** — read the board: - **More than one left** → design the single most **discriminating** experiment: one the surviving suspects *predict differently*. Run it; it eliminates at least one. Loop. - **Exactly one left** → that is the truth, **however improbable**. *Confirm* it (toggle it on/off and watch the symptom follow; fix it and watch it vanish), then fix. Verify before you claim. - **Zero left** → **stop and distrust your eliminations.** Find the suspect you struck off on *assumption* rather than *proof* and re-test it, or add the improbable cause you never listed. (E.g. a "deterministic / no shared state" alibi is an *assumption* — prove determinism with two byte-identical runs and bisect the suite to expose state a prior test leaked.) Nothing-remains means the method was misapplied, not that the bug is supernatural. Full detail (the bisection strategy, what counts as proof, the over-eliminate / under-enumerate failure modes, discriminating-test design): **`references/elimination-method.md`**. ## Mental model: differential diagnosis on a case board - **It's a whodunit, and you're building the suspect board** — names pinned up, each crossed off only when it has an alibi *backed by evidence*. - **An alibi is proof of impossibility, not a character reference.** "She's a trusted library" is not an alibi. "The stale response carries an old build-SHA" is. - **The remaining suspect is guilty even if they have no motive.** Improbability is not innocence. Confirm, then convict. - **If everyone has an alibi, one alibi is forged** — or the culprit was never on the board. Re-examine the alibis; widen the list. ## Worked example: the API that serves stale data "impossibly" **Symptom:** ~3% of requests return data ~5 minutes stale. "Impossible — we invalidate the cache on every write, and the old version was decommissioned." 1. **OBSERVE.** Hammer the endpoint; capture response headers per request. **Negative fact (dog didn't bark):** the `cache.invalidate` log line is *absent* on the stale responses — the invalidation path didn't run for them. 2. **ENUMERATE** (complete, incl. the "impossible"): (a) app in-process cache TTL, (b) shared Redis cache stale, (c) CDN/edge cache, (d) DB read-replica lag, (e) a **zombie old-version instance** still in the LB pool ("but it's decommissioned"), (f) client-side cache, (g) clock skew breaking TTL math ("but NTP-synced"), (h) the load test reusing a cached HTTP client. 3. **ELIMINATE by proof** (not by reputation): - (c) CDN: stale responses carry `X-Cache: MISS`, `Cache-Control: no-store` → can't be the CDN. ✗ - (f) client / (h) harness: reproduced with a fresh `curl --no-keepalive` → ✗ both. - (d) replica lag: the stale value predates the last write by *minutes*, but replica-lag metric is <1s → ✗. - (b) Redis: a direct `GET` of the key returns the *fresh* value with a correct TTL → ✗. - (g) skew: host clocks agree within 5 ms → ✗. - (e) "decommissioned instance": **do not eliminate on assumption.** Test it — stamp every response with its instance build-SHA. 3% of stale responses carry an **old SHA**. 4. **CONVERGE.** One survivor: (e). However improbable ("it's decommissioned"), it's the truth — a half-finished deploy left a zombie instance in the load balancer serving old code, and its invalidation path was the old one (hence the silent dog). **Confirm:** drain that instance → stale rate drops to 0, no more old SHAs. Fix the deploy to deregister it. **VERIFY BEFORE CLAIM:** paste "stale_rate 0/100k, zero old SHAs," not "should be fixed." The lesson is the 0-survivor trap dodged: had you struck (e) off on "it's decommissioned," your board would be empty and you'd be "stuck on an impossible bug" — when really you'd eliminated the truth on an assumption. ## Quick start 1. **Open a case board** — write the exact symptom and the facts, including the *negative* facts (what didn't happen): ```bash cp skills/sherlock/templates/suspect-board.md suspect-board.md ``` 2. **Enumerate the COMPLETE suspect list** — sweep every layer (code · config · data · build · deps · runtime · hardware · env · concurrency · state · the harness itself) and **include the improbable**. Probability comes later. 3. **For each suspect, write the decisive falsifying test** — the cheapest observation that proves it *impossible*. Run the most **bisecting** one first; strike a suspect only on **evidence**, and record *how*. 4. **Converge:** >1 left → run the most **discriminating** experiment; exactly 1 → **confirm the improbable survivor** and fix; 0 → **un-eliminate the assumption** or widen the list. Then verify the fix with the real artifact. ## References - `references/elimination-method.md` — the loop in depth: building a *complete* differential, what counts as proof (eliminate-by-evidence vs eliminate-by- assumption), bisecting the suspect set, designing the discriminating experiment, and the over-eliminate / under-enumerate branches when nothing remains. - `references/holmesian-principles.md` — the Holmes canon mapped to debugging, operationally: data-before-theory, observe-vs-see, the dog in the night-time (negative evidence), reasoning backward, twist-theories-to-facts, and the maxim itself — each with the bug-hunt move it licenses. - `references/biases-and-false-eliminations.md` — why you wrongly strike off the true cause (incredulity, confirmation bias, anchoring, recency/"blame the last change", authority/"it's well-tested"), the catalogue of "impossible" culprits that are actually possible (your UB not the compiler; your misuse not the stdlib; the test harness; clock skew; caches; a zombie instance; ECC/cosmic-ray), and the rationalization table + red flags. ## Templates - `templates/suspect-board.md` — a fill-in case board: the symptom + facts (incl. negative facts), the complete suspect list across layers, and per suspect a status (open / eliminated-by-evidence / **surviving**), the decisive test, and the evidence — so the investigation is complete, auditable, and never re-litigates a cleared suspect. ## Relationship to systematic-debugging, andrej & brendan Sherlock is the **elimination lens**, not a replacement for the broader debugging flow: - **superpowers:systematic-debugging** is the overall reproduce → isolate → root-cause loop. Sherlock is what you run **inside the isolate/root-cause step** when many causes are in play and the obvious one is gone. Use systematic-debugging for the flow; reach for Sherlock the moment you're *choosing among suspects*. - **`andrej`** is the observe→analyze→act metric loop: it makes the symptom observable (the instrument-first step in *When NOT to use*) and then carries 1–2 hypotheses, iterating each toward a metric. Sherlock instead enumerates **all** hypotheses and kills them. They meet at the discriminating experiment: Sherlock decides *which* one-variable test best separates the survivors; `andrej` runs it cleanly (fixed budget, keep-or-revert) and logs it. Use Sherlock to pick the experiment; `andrej` to execute it. - **`brendan`** is elimination specialized to **performance** — the USE sweep enumerates resources and rules each out. Sherlock is the general-purpose version for **correctness** bugs; the shape (complete list → eliminate each → the survivor is it) is identical.