Wrong-Number Debugging
Overview
A surprising number is data trying to tell you something. The instinct is to patch it — add a dropna, a distinct, a filter — until it looks reasonable. That instinct is how a symptom gets hidden and the real bug ships. Routing: code THROWS or a test fails → superpowers:systematic-debugging. Code runs CLEAN but the number is wrong → this skill. Either way, a remedy that changes design/sample/spec is analysis-checkpoints territory, not a fix.
Core principle: Locate the bug by bisecting the pipeline, not by guessing at fixes. The number is wrong somewhere specific — find where, then you'll know why.
Why analytics debugging is its own thing
In software a bug announces itself with a stack trace pointing near the cause. In analysis there is no trace — the pipeline ran clean, and the only signal you have is that the output is wrong. Work backward through the chain of joins, filters, groupings, and recodes, checking the number at each stage, until you find where it stopped being right. That stage contains the bug.
The loop
REPRODUCE → LOCATE (bisect) → EXPLAIN → FIX AT THE SOURCE → RE-CONTRACT
REPRODUCE — Pin the wrong number down to a deterministic, minimal case. Same input, same seed, same result every time. If it's intermittent, you have hidden state (ordering, randomness, a mutated global) and that is the bug. Shrink to the smallest subset of rows that still shows it — debugging on 50 rows beats debugging on 50 million.
Then state the diagnostic roadmap and get a quick nod before running scans. A bisection is a multi-step plan — state it in 2–4 lines: the stages you'll check, in what order, where you'll start. "Roadmap: (1) pull the flagged records, (2) check X, (3) scan the panel for the pattern, (4) trace how it's produced. Stays a diagnosis — any drop/merge comes back to you. Good, or reprioritize?" Get agreement once, then execute autonomously — only re-stopping if a step turns into a design/sample/spec change. This is where the user's local knowledge reorders your search cheaply. Skip it for a one-or-two-step check.
LOCATE by bisection — Execute the agreed roadmap. Walk the pipeline and check the number at each intermediate stage:
- What is the row count / total / value right after load? Is it already wrong? Then the bug is upstream — in the source or the extract, not your code.
- After each join? (Joins are the prime suspect — check row counts before and after every one.)
- After each filter? (Did a filter on a column with
NA/missing silently drop rows you wanted? Null-aware filters surprise people in every language.)
- After each group-by / aggregate? (Wrong grain, double-counting, a
sum over a fanned-out join.)
- After each recode / type cast? (A coercion turned
"1,000" into NA, or strings into a factor with a surprise level.)
Binary-search it: check the middle of the chain. Wrong already? Bug is in the first half. Still right? Second half. A ten-step pipeline localizes in three or four checks.
EXPLAIN — Once you've found the stage, explain why in one sentence before touching code. "The join fanned out because customer_id is not unique in the orders table." If you can't articulate the mechanism, you haven't found the bug yet — keep bisecting. A patch that merely makes the number look right is more dangerous than the obviously-wrong one it replaced — the bug is now hidden, not fixed.
FIX AT THE SOURCE — but first answer the dividing question: am I restoring the analysis we agreed on, or changing it? Answer it before you touch anything; the two branches are very different:
- Data-bug fix (restores the intended computation): the join fanned out, so dedup the right table or aggregate before joining; the units were wrong, so correct them; the date parse broke, so repair it. This returns the analysis to what was already agreed — do it, then report what you found and fixed. (Don't slap a
distinct() on the final output to paper over tripled rows — fix the key, not the symptom.)
- Analytical-design change (changes what is being estimated): the bug is real, but the remedy would change the design, the spec, the sample, or the estimand. This is not yours to do — STOP and route it through
analysis-checkpoints. Present the threat, the candidate remedies, and your recommendation, and let the user decide. Implementing the redesign and presenting it as "the fix" is exactly the behind-the-back decision to avoid.
- The trap: winsorizing, dropping outliers, adding a control, or restricting the sample feels like cleaning but is a sample/spec change — it belongs on the design side (STOP), not the data-bug side. And a genuine data-bug fix that nonetheless moves a number the user has already seen still gets surfaced before you build on it.
RE-CONTRACT — Add a data-contracts check that would have caught this, and watch it bite on the broken version. A bug you found once should never silently return. (If the user approved a redesign in the branch above, the new spec re-enters at REPRODUCE — it's a new result to validate, not a closed bug.)
Trace provenance backward — the usual culprits
The categories worth checking at each bisected stage are the same ones data-contracts' invariant catalog asserts and result-verification's checklist reconciles — don't re-derive the list, bisect against it: fan-out join (totals inflated by a clean-ish 2×/3×; check row counts and key uniqueness), vanishing rows (an inner join or null-dropping filter silently discarded unmatched rows; anti-join to see what failed to match), NA/missing poisoning, units/scale (dollars vs. cents, ms vs. s), wrong grain (double-counting user-sessions as users), surprise category level (a stray "unknown", mojibake, a trailing-space duplicate), temporal (a timezone shift or resample duplicating a period), and upstream surgery (the data was already sampled or filtered before it reached you).
Language cheat-sheet
| Need |
Python |
R |
Julia |
| Row count at a stage |
len(df) / df.shape[0] |
nrow(df) |
nrow(df) |
| See unmatched join keys |
merge(..., indicator=True) then count |
anti_join(a, b) |
antijoin(a, b, on=:id) |
| Count missing |
df.isna().sum() |
colSums(is.na(df)) |
count(ismissing, col) |
| Distinct levels |
df.col.value_counts(dropna=False) |
table(df$col, useNA="always") |
countmap(col) |
| Key uniqueness |
df.id.is_unique |
!any(duplicated(df$id)) |
allunique(df.id) |
Red flags — STOP
- Adding a
dropna / distinct / filter to make a number look right, without knowing why it was wrong.
- "It's probably just duplicates" — without having checked key uniqueness.
- Debugging on the full dataset instead of shrinking to a minimal failing case.
- Fixing the final output when the bug is three joins upstream.
- Declaring it fixed without adding a check that bites on the old broken version.
- Changing the research design, spec, sample, or estimand to make a number behave — without surfacing it to the user as their decision (
analysis-checkpoints).
Common rationalizations
| Excuse |
Reality |
"A distinct() makes it match, good enough." |
If you don't know which rows were duplicated and why, you don't know what else that distinct is silently dropping. |
| "It's a small discrepancy, probably rounding." |
"Small" discrepancies are often a few leaking rows. Find the rows before you blame the floats. |
| "I'll just rebuild the query from scratch." |
You'll likely reintroduce the same bug. Localize it first; understand it; then rebuild if you must. |
| "The number looks reasonable now." |
"Looks reasonable" is the exact disguise a hidden bug wears. Reasonable ≠ reconciled. |
The Process
- Reproduce minimally, then state the diagnostic roadmap (stages, order, where you start) and get a quick nod — a real bisect is a multi-step plan the user should be able to reorder before you run scans; agree once, then execute autonomously. Name the mechanism in one sentence. No fix until you can.
- Answer the dividing question — restoring the agreed analysis, or changing it?
- Data bug → fix at the source, then invoke
data-contracts to add the invariant that would have caught it and watch it bite on the broken version — never patch the symptom on the final output.
- *Design/sample/spec/estimand change → STOP and invoke
analysis-checkpoints.* Present the threat, the candidate remedies, and your recommendation; do not smuggle a redesign in as a bug fix. An approved redesign re-enters at REPRODUCE as a new result to validate.
- Log the lesson while the bisect is fresh — one line to the project's
docs/LESSONS.md (symptom, cause, the check that would have caught it; create the file if absent). You just paid for the exact failure class; thirty more seconds converts it into a standing check instead of a future re-bisect. If it would bite any analysis, propose folding it into the relevant skill (result-verification → "Capture what bit you" — skill edits need sign-off).
The bottom line
Wrong number → reproduced minimally, bisected to the stage, mechanism named, fixed at the source, check added
Otherwise → a symptom patched and a bug still in the pipeline
1---2name: wrong-number-debugging3description: Use the moment a computed result looks wrong, surprising, suspicious, or "off" — a total that doesn't reconcile, revenue that tripled after a join, a mean that moved, a coefficient with the wrong sign, a count that's too high or too low, a metric that disagrees with another team's. Bisects the data pipeline to find the exact step where the number went bad, tracing provenance backward instead of guessing-and-patching. Use whenever the user says "this number looks wrong", "why is this so high", "these don't match", "the totals are off", "that can't be right", or when a `data-contracts` reconciliation fails — in R, Julia, Python, or Stata.4---56# Wrong-Number Debugging78## Overview910A surprising number is data trying to tell you something. The instinct is to patch it — add a `dropna`, a `distinct`, a filter — until it looks reasonable. That instinct is how a symptom gets hidden and the real bug ships. **Routing:** code THROWS or a test fails → `superpowers:systematic-debugging`. Code runs CLEAN but the number is wrong → this skill. Either way, a remedy that changes design/sample/spec is `analysis-checkpoints` territory, not a fix.1112**Core principle:** Locate the bug by bisecting the pipeline, not by guessing at fixes. The number is wrong *somewhere specific* — find where, then you'll know why.1314## Why analytics debugging is its own thing1516In software a bug announces itself with a stack trace pointing near the cause. In analysis there is no trace — the pipeline ran clean, and the only signal you have is that the *output* is wrong. Work backward through the chain of joins, filters, groupings, and recodes, checking the number at each stage, until you find where it stopped being right. That stage contains the bug.1718## The loop1920```21REPRODUCE → LOCATE (bisect) → EXPLAIN → FIX AT THE SOURCE → RE-CONTRACT22```23241. **REPRODUCE** — Pin the wrong number down to a deterministic, minimal case. Same input, same seed, same result every time. If it's intermittent, you have hidden state (ordering, randomness, a mutated global) and *that* is the bug. Shrink to the smallest subset of rows that still shows it — debugging on 50 rows beats debugging on 50 million.2526 **Then state the diagnostic roadmap and get a quick nod before running scans.** A bisection is a multi-step plan — state it in 2–4 lines: the stages you'll check, in what order, where you'll start. *"Roadmap: (1) pull the flagged records, (2) check X, (3) scan the panel for the pattern, (4) trace how it's produced. Stays a diagnosis — any drop/merge comes back to you. Good, or reprioritize?"* Get agreement once, then execute autonomously — only re-stopping if a step turns into a design/sample/spec change. This is where the user's local knowledge reorders your search cheaply. Skip it for a one-or-two-step check.27282. **LOCATE by bisection** — Execute the agreed roadmap. Walk the pipeline and check the number at each intermediate stage:29 - What is the row count / total / value right after **load**? Is it already wrong? Then the bug is upstream — in the source or the extract, not your code.30 - After each **join**? (Joins are the prime suspect — check row counts before and after every one.)31 - After each **filter**? (Did a filter on a column with `NA`/`missing` silently drop rows you wanted? Null-aware filters surprise people in every language.)32 - After each **group-by / aggregate**? (Wrong grain, double-counting, a `sum` over a fanned-out join.)33 - After each **recode / type cast**? (A coercion turned `"1,000"` into `NA`, or strings into a factor with a surprise level.)3435 Binary-search it: check the middle of the chain. Wrong already? Bug is in the first half. Still right? Second half. A ten-step pipeline localizes in three or four checks.36373. **EXPLAIN** — Once you've found the stage, explain *why* in one sentence before touching code. "The join fanned out because `customer_id` is not unique in the orders table." If you can't articulate the mechanism, you haven't found the bug yet — keep bisecting. A patch that merely makes the number *look* right is more dangerous than the obviously-wrong one it replaced — the bug is now hidden, not fixed.38394. **FIX AT THE SOURCE — but first answer the dividing question:** *am I restoring the analysis we agreed on, or changing it?* Answer it before you touch anything; the two branches are very different:40 - **Data-bug fix** (restores the intended computation): the join fanned out, so dedup the right table or aggregate before joining; the units were wrong, so correct them; the date parse broke, so repair it. This returns the analysis to what was already agreed — **do it, then report what you found and fixed.** (Don't slap a `distinct()` on the final output to paper over tripled rows — fix the key, not the symptom.)41 - **Analytical-design change** (changes what is being estimated): the bug is real, but the remedy would change the design, the spec, the sample, or the estimand. **This is not yours to do — STOP and route it through `analysis-checkpoints`.** Present the threat, the candidate remedies, and your recommendation, and let the user decide. Implementing the redesign and presenting it as "the fix" is exactly the behind-the-back decision to avoid.42 - **The trap:** *winsorizing, dropping outliers, adding a control, or restricting the sample* feels like cleaning but is a **sample/spec change** — it belongs on the design side (STOP), not the data-bug side. And a genuine data-bug fix that nonetheless *moves a number the user has already seen* still gets surfaced before you build on it.43445. **RE-CONTRACT** — Add a `data-contracts` check that would have caught this, and watch it bite on the broken version. A bug you found once should never silently return. (If the user approved a *redesign* in the branch above, the new spec re-enters at REPRODUCE — it's a new result to validate, not a closed bug.)4546## Trace provenance backward — the usual culprits4748The categories worth checking at each bisected stage are the same ones `data-contracts`' invariant catalog asserts and `result-verification`'s checklist reconciles — don't re-derive the list, bisect against it: **fan-out join** (totals inflated by a clean-ish 2×/3×; check row counts and key uniqueness), **vanishing rows** (an inner join or null-dropping filter silently discarded unmatched rows; anti-join to see what failed to match), **NA/missing poisoning**, **units/scale** (dollars vs. cents, ms vs. s), **wrong grain** (double-counting user-sessions as users), **surprise category level** (a stray `"unknown"`, mojibake, a trailing-space duplicate), **temporal** (a timezone shift or resample duplicating a period), and **upstream surgery** (the data was already sampled or filtered before it reached you).4950## Language cheat-sheet5152| Need | Python | R | Julia |53|---|---|---|---|54| Row count at a stage | `len(df)` / `df.shape[0]` | `nrow(df)` | `nrow(df)` |55| See unmatched join keys | `merge(..., indicator=True)` then count | `anti_join(a, b)` | `antijoin(a, b, on=:id)` |56| Count missing | `df.isna().sum()` | `colSums(is.na(df))` | `count(ismissing, col)` |57| Distinct levels | `df.col.value_counts(dropna=False)` | `table(df$col, useNA="always")` | `countmap(col)` |58| Key uniqueness | `df.id.is_unique` | `!any(duplicated(df$id))` | `allunique(df.id)` |5960## Red flags — STOP6162- Adding a `dropna` / `distinct` / `filter` to make a number look right, without knowing why it was wrong.63- "It's probably just duplicates" — without having checked key uniqueness.64- Debugging on the full dataset instead of shrinking to a minimal failing case.65- Fixing the final output when the bug is three joins upstream.66- Declaring it fixed without adding a check that bites on the old broken version.67- **Changing the research design, spec, sample, or estimand to make a number behave — without surfacing it to the user as their decision (`analysis-checkpoints`).**6869## Common rationalizations7071| Excuse | Reality |72|---|---|73| "A `distinct()` makes it match, good enough." | If you don't know which rows were duplicated and why, you don't know what else that `distinct` is silently dropping. |74| "It's a small discrepancy, probably rounding." | "Small" discrepancies are often a few leaking rows. Find the rows before you blame the floats. |75| "I'll just rebuild the query from scratch." | You'll likely reintroduce the same bug. Localize it first; understand it; then rebuild if you must. |76| "The number looks reasonable now." | "Looks reasonable" is the exact disguise a hidden bug wears. Reasonable ≠ reconciled. |7778## The Process79801. **Reproduce minimally, then state the diagnostic roadmap (stages, order, where you start) and get a quick nod** — a real bisect is a multi-step plan the user should be able to reorder before you run scans; agree once, then execute autonomously. Name the mechanism in one sentence. No fix until you can.812. **Answer the dividing question** — restoring the agreed analysis, or changing it?823. **Data bug → fix at the source, then *invoke `data-contracts`*** to add the invariant that would have caught it and watch it bite on the broken version — never patch the symptom on the final output.834. **Design/sample/spec/estimand change → STOP and *invoke `analysis-checkpoints`*.** Present the threat, the candidate remedies, and your recommendation; do not smuggle a redesign in as a bug fix. An approved redesign re-enters at REPRODUCE as a new result to validate.845. **Log the lesson while the bisect is fresh** — one line to the project's `docs/LESSONS.md` (symptom, cause, the check that would have caught it; create the file if absent). You just paid for the exact failure class; thirty more seconds converts it into a standing check instead of a future re-bisect. If it would bite *any* analysis, propose folding it into the relevant skill (`result-verification` → "Capture what bit you" — skill edits need sign-off).8586## The bottom line8788```89Wrong number → reproduced minimally, bisected to the stage, mechanism named, fixed at the source, check added90Otherwise → a symptom patched and a bug still in the pipeline91```
lancegui