Result Verification

Result Verification

Overview

This skill is user-invoked. When a result is about to ship unverified, offer verification in one line — never run it unasked (and never widen it: see below).

The last mile is where good analyses die. The number is computed, it looks right, the deadline is close — and "looks right" becomes "is right" without anything in between. This skill is the something in between: the checks that stand between a computed number and a claimed result.

This is the analytics counterpart of verification-before-completion. The rule is identical: evidence before assertions, always. You do not say "the result is X"; you say "the result is X, here is the reconciliation, here is the clean-room reproduction, here is what the approved robustness showed." Verification confirms; it never widens — running a spec, cut, or placebo nobody approved is not extra diligence, it's an unapproved analysis decision (analysis-checkpoints).

Core principle: A result is not done when it appears; it's done when it has been reconciled, reproduced from scratch, and survived being attacked.

The verification checklist

Run these before any result leaves your hands. Each maps to a real way final numbers turn out wrong:

1. Reconcile to source. Do the parts sum back to the known whole? Does the headline number tie out to a total you can compute a completely different way? A revenue figure should reconcile to the raw ledger; a user count to a distinct count of IDs. Reconciliation by an independent path is the single strongest check that the number is real. (Use float-aware comparison.)

2. Reproduce from a clean state. Restart the kernel / session / R process — no cached objects, no leftover variables — set the seed, and run the analysis end to end from raw inputs. A result that only exists because of a variable still in memory from three hours ago is not a result. If it doesn't reproduce, you don't have a finding, you have an artifact.

3. Confirm determinism. Same input + same seed → same output, twice. If the number wiggles between runs, there's uncontrolled randomness or ordering dependence, and the figure you're about to report is one sample from a distribution you didn't mean to draw from.

4. Confirm the robustness that already ran — never initiate robustness here. For confirmatory work, confirm the suite pre-committed in pre-analysis-plan ran — all of it. Otherwise, if the user approved a robustness shortlist (causal-identification, analysis-checkpoints), confirm it ran and what it showed; if none was approved, verification proceeds without one — proposing robustness is an analysis decision that belongs upstream with the user, not a verification step. If the headline swings wildly under a reasonable perturbation, it is fragile and you must say so. Confirm each variant actually moved something: a robustness check that returns a number identical to the baseline didn't perturb anything — a "leave-one-out" that re-added the row, a subsample filter that matched everything, a flag never read — and is a silent no-op, not evidence of robustness. A check has to bite to count.

5. Read it like an economist — interpretable units, economic (not just statistical) significance, magnitude plausibility, mechanism, and a benchmark against known estimates. This is the heart of verification for any effect you'll interpret; the full discipline is below.

6. Tie the artifacts to the prose. Every number in the text, every figure axis, every table cell — does it match what the code actually produced in this run? Stale numbers from an earlier version, a figure that wasn't regenerated, a rounded value that contradicts the table: these are the embarrassing errors that survive everything else because nobody re-checked the copy against the output.

7. For a structural estimate, verification carries extra load-bearing checks beyond reconcile-and-reproduce: the Monte-Carlo recovery passed (the estimator recovers known θ from a distant start); the model fits moments it wasn't targeted on and holds up out-of-sample; the counterfactual was computed by re-solving equilibrium, not holding endogenous objects fixed; and the implied object (e.g. the own-price elasticity at observed prices) reconciles with the raw descriptive picture. A counterfactual that only exists because prices were held fixed is not verified. See structural-estimation.

8. For a confirmatory result, reconcile against the pre-analysis plan. Was the reported number the pre-committed primary spec, interpreted against the PAP's decision rule, with secondary tests corrected as pre-registered? Report a pre-registered null as a finding — don't bury it (pre-analysis-plan).

9. Get an independent pass, then tidy — before you report. Dispatch the analysis-reviewer agent for a fresh-context review of your own work (it catches what you rationalized), and tidy the workspace so what ships is deliverables, not scratch (project-organization). These are steps of verification, not optional extras the user has to ask for.

Reliability is not validity — verify the number means what you think

Reconcile, reproduce, determinism, robustness establish reliability — the number is computed correctly from the data. They do not establish validity — that the quantity measures the construct you named. A figure can tie out to source, reproduce from a clean session, and survive every robustness re-cut, and still be a precise measure of the wrong thing: a count of visible traces reported as a count of the underlying behavior, a proxy label treated as the truth, a coverage-limited slice named as the whole. Every internal check passes; the number is reliable and wrong.

Validity is checked against something outside the dataset, and at least one such check belongs in verification whenever the number's level (not just its precision) carries the claim — for a descriptive count or rate as much as for an estimated effect:

• A known shock that should move it — does the series respond, at the right time and sign, to an event that should change it?
• An external benchmark — does the level sit in a defensible range next to an independent estimate of the same quantity? An order-of-magnitude gap is a finding to explain with a mechanism, not to report flat.
• Alternative-construct coverage — expand the definition along the dimension you suspect is missing and see how far the level moves; a level that is an artifact of where you looked is a coverage limit, not a fact.

The deep validity craft for a descriptive count lives in descriptive-evidence (plausibility triangulation); for an effect it's the magnitude / mechanism / benchmark pass below. A number that clears every reliability check but no validity check is submitted, not verified.

A check failed — stop, don't ship behind a caveat

If a check here fails and you cannot resolve it — a total won't reconcile, the estimate swings wildly under a reasonable perturbation, the magnitude is absurd — stop and bring the failure to the user as a decision. Do not report the result anyway with the problem buried in a caveat; "evidence before assertion" means a failed check blocks the claim, not footnotes it. (Where the fix is a data bug, route to wrong-number-debugging; where it's a design/sample/spec change, to analysis-checkpoints.)

Read the estimate like an economist

A coefficient that reconciles and reproduces can still be economically meaningless or absurd. Reproducibility tells you the number is real; this tells you whether it's believable and whether it matters:

• Convert to interpretable units. A raw coefficient is not yet a finding. Turn it into an elasticity, a semi-elasticity, a percent of the mean, a fraction of an SD, a dollar figure — whatever lets a reader feel the size. "0.043" means nothing; "a 4% increase, or about a third of the control-group gap" means something.
• Economic significance, not just statistical. The question is never only "is it distinguishable from zero" — it's "is it big enough to matter for the decision or for welfare." A precisely-estimated tiny effect and a precise zero are, economically, the same answer: no. Say so, rather than dressing a trivial effect in stars.
• Back-of-envelope the magnitude. Does the size survive contact with how the world works? Translate it into an implied behavioral response, an implied total dollar amount, or an implied share of a known aggregate, and check that the implication isn't absurd (an effect larger than the outcome's possible range, or a dollar figure exceeding the whole market).
• Mechanism consistency. Does the sign and size match the channel you posited in question-framing? If the mechanism has auxiliary predictions (it should bite harder for some subgroup, show up in an intermediate outcome), check those too — a real effect usually leaves more than one fingerprint.
• Benchmark against what's known. How does it compare to existing estimates of the same or similar parameter? Being far off the literature isn't disqualifying, but it demands an explanation you can state. An estimate 10× the consensus is a claim that you've overturned the consensus — be sure that's what you mean.

When the magnitude is implausible, that is a result to investigate (wrong-number-debugging) or a finding to defend with a mechanism — never a number to report with a shrug.

Evidence before assertion

The failure mode is claiming completion you haven't verified — "the analysis is done," "the numbers check out," "it reproduces." Replace every such claim with the output that proves it. Don't write "totals reconcile"; show the reconciliation line where the two independently-computed numbers match. Don't write "it's robust"; show the table of the estimate under each perturbation. An unverified "done" is just hope with a deadline.

Freeze the verified result

Once it passes, snapshot it as a golden output (see data-contracts). The verified number becomes the baseline that the next run is diffed against — so if a refactor or a data refresh silently changes it, you find out loudly instead of three weeks later in a meeting.

Consult — and capture — what bit you

Consult first (at the start, and before you report). An established project keeps its scar tissue in docs/LESSONS.md and in your memory — the silent failures that bit it before. Read them when you pick up the analysis and again before reporting, and check whether any apply here: a prior fan-out on these tables, a vintage mismatch in this geography, a figure-vs-note estimand trap. (These are recalled, not folded wholesale into this skill: the lesson is domain-specific and lives in the project.)

Capture at the end. Before you close out, do a 60-second retro: what silent failure actually bit this project — the fan-out join, the leaked feature, the bad control, the implausible magnitude you almost shipped? Write it down in docs/LESSONS.md (one line: symptom, cause, the check that would have caught it). A lesson recorded is a bug that won't recur silently.

Folding a lesson into the general skills is the rare exception, not the default. Most lessons are domain-specific (this dataset, this geography) and belong only in the project — reached by the consult step above, never dragged into the shared family. Fold one upward only when the pattern, stripped of domain, would help a project that's never seen this data (e.g. "versioned join keys need a vintage assertion"); leave the instance in the project, and skill edits need the user's sign-off — propose, don't silently rewrite.

And keep the stores lean. If, while consulting, you find LESSONS.md sprawling or a memory file grown into a document, suggest a consolidation/prune pass (the consolidate-memory skill for memory) — surface it, don't hoard, don't auto-run.

Language cheat-sheet

Red flags — STOP

• About to report a number you computed but never reconciled by a second path.
• "It reproduces" — but you never actually restarted the kernel and re-ran from raw inputs.
• A figure or table number that you haven't confirmed matches the current run's output.
• Reporting a point estimate with no idea whether it survives dropping a few outliers.
• A magnitude — especially one that surprised you, high or low — never checked against anything external (a known shock, a benchmark, a wider definition).
• Writing "the numbers check out" instead of showing the output that checks them out.
• Verification that quietly grows extra specifications — a subsample re-estimate, a leave-one-out, a placebo — that no one approved. Propose them if you think they're needed; never run them on your own.

Common rationalizations

The Process

1. Run the checklist — reconcile to source, reproduce clean, confirm determinism, confirm the approved robustness, read it like an economist, tie artifacts to prose, freeze the golden output.
2. If any check fails and you can't resolve it → STOP and invoke wrong-number-debugging to bisect (or analysis-checkpoints if the fix is a design/sample/spec change). Do not footnote a failed check.
3. All checks pass → invoke analysis-review — dispatch the independent analysis-reviewer to catch the silent failures you rationalized, before you ship.
4. Then invoke project-organization — tidy scratch from deliverables before you commit or push. Don't end at "the result is X" — route to the review and the tidy-up.

The bottom line

Reported result  →  reconciled by an independent path, reproduced from a clean state, approved robustness confirmed (if any), artifacts tied to prose
Otherwise         →  not verified, just submitted