Audit Reproducibility

Audit Reproducibility

Compare numeric claims in a manuscript (point estimates, standard errors, p-values, counts) against the actual outputs produced by the analysis pipeline. Report PASS / FAIL per claim against the tolerance thresholds defined in .claude/rules/replication-protocol.md.

Core principle: If the paper says ATT = -1.632 (0.584) and the code produces -1.628 (0.591), we verify — numerically — that the difference is within the documented tolerance. No more "looks close enough" eyeballing.

Two directions, not one. Vertically, each claim is checked against the output that produced it. Horizontally, it is checked against every other artifact that displays the same number — the supplement table, the slide deck, the poster. The vertical check passes contentedly while a deck quotes last month's value; only the horizontal one catches that. Declared displays live in the passport's appears_in list — see replication-protocol.md → The horizontal check.

When to use

• Before submission. Catches the "I updated the analysis but forgot to update Table 2" bug.
• Before presenting or teaching from the same numbers. Catches the deck, the poster, or the supplement that was never regenerated after the last rerun.
• Before releasing a replication package. Verifies the code actually reproduces the paper.
• After a major revision. Ensures the paper still matches the latest code.
• Quality-gate in /commit. Pair with a pre-commit invocation on manuscript + analysis changes.

Inputs

• $0 — path to the manuscript (.tex, .qmd, .md, .pdf). Required.
• $1 — path to the outputs directory. Defaults to scripts/R/_outputs/. Recognised alternatives: scripts/stata/_outputs/ (Stata pipelines built by /stata-replication), _targets/objects/ (R targets workflows), any directory the user-specified outputs live in.

Workflow

Phase 0: Pre-flight

1. Read replication-protocol.md for the tolerance thresholds currently in effect.
2. Verify the outputs directory exists and is non-empty. If empty or stale (older than the manuscript), prompt the user to re-run their pipeline (e.g., Rscript scripts/R/00_run_all.R) before auditing.
3. Ensure a sessionInfo.txt or equivalent environment capture exists in the outputs dir.

Phase 1: Extract claims from the manuscript

Parse the manuscript for numeric claims. Patterns to match:

• Point-estimate + SE: ATT = -1.632 (0.584), $\beta = 0.342$ (0.091), hat{\tau} = 1.28** with starred significance
• Table cells: & -1.632$^{***}$ & 0.584 & in LaTeX table environments
• Counts: our sample of 2,847 firms, $N = 2{,}847$
• Summary stats: mean = 0.423, SD = 0.087
• P-values: p < 0.01, $p = 0.003$

Record each claim as a tuple:

{
  claim_id: "Table2_col3_ATT",
  location: "Table 2, Column 3, row 'Treatment'",
  kind: "point_estimate" | "standard_error" | "p_value" | "count" | "percentage",
  reported_value: -1.632,
  uncertainty: 0.584,              # only for point estimates
  significance_stars: 3,            # 0-3 or None
  raw_context: "the ATT estimate of -1.632 (0.584) indicates..."
}

Write the extracted claims to quality_reports/reproducibility_claims_[manuscript-name].json so the user can review the extraction before audit.

Phase 2: Extract results from outputs

Scan $1 for corresponding values. Priority order:

1. .rds files — readRDS(path)$coef[["treatment"]] style lookups. Can use Rscript -e "saveRDS(summary(readRDS(...)), '/tmp/audit.rds')" to extract.
2. .tex tables — parse LaTeX table cells directly; match on column headers + row labels.
3. .csv summary files — pandas/readr parse, key-value lookup.
4. .out / .log files (Stata, regress output) — regex extraction.
5. .json — direct key lookup.

Record each extracted result:

{
  source: "scripts/R/_outputs/results.rds",
  lookup_key: "fit_main$coefficients['treated']",
  value: -1.628,
  uncertainty: 0.591,
  p_value: 0.005
}

Phase 2b: Collect every declared display (passport mode)

For each passport claim, read its appears_in list and pull the value as displayed at each entry:

{
  claim_id: "C3",
  displays: [
    { path: "manuscript.tex", locator: "Table 1, Col 2", display_precision: 3, shown: 0.342 },
    { path: "Slides/Lecture04_Results.tex", locator: "frame 'Main result'", display_precision: 2, shown: 0.34 }
  ]
}

Rules for this pass:

• The primary location: is one of the displays, not a separate thing — if appears_in repeats it, that is one display, not two.
• A declared display that cannot be resolved — missing file, or no value at the locator — is recorded as shown: NOT_FOUND. It resolves to FAIL in Phase 4c. Never skip it: a locator that has quietly stopped matching is exactly where a stale number hides.
• A claim with no appears_in list is horizontally unchecked, not horizontally clean. Report it that way (see Phase 5) rather than silently passing it.

Phase 3: Match claims to results

Use fuzzy heuristics when exact labels don't match:

• Name similarity ("treatment effect" ~ "ATT" ~ "treated")
• Magnitude similarity (if two candidates have values within 10% of the reported, prefer the one with closer SE)
• Context hints from the claim's raw_context field (table number, row label, description)

For every claim, produce a match candidate with a confidence score. Claims below 0.7 confidence get flagged as "UNMATCHED — manual review needed" rather than silently passing.

Phase 4: Tolerance check

For each matched claim, apply the thresholds from replication-protocol.md:

Respect any tolerance overrides the user has written into their replication-protocol.md fork (they may loosen for MC noise or tighten for administrative data).

Phase 4b: Disposition — PASS / FAIL / EXPLAINED / UNMATCHED

A tolerance check resolves to one of four dispositions:

• PASS — within tolerance.
• FAIL — outside tolerance, with no defensible alternative recorded. Blocks (exit 1).
• EXPLAINED — outside tolerance, but the author has recorded a concrete, named alternative specification that accounts for the gap (see the downgrade rule). Surfaced in the report and carried into a response-to-referees; does not block.
• UNMATCHED — no computed counterpart found (Phase 3 confidence < 0.7). Never auto-downgradable.

A mismatch is not automatically a failure. In applied work the most common out-of-tolerance result is a defensible alternative spec, not a bug — reghdfe vs feols clustering df, a different bandwidth-selection rule, a different MC seed/reps, or display rounding. The skill's job is to stage the disagreement for a human auditor, not to pronounce the code right and the paper wrong. (The df-adjustment note in "Stata-specific notes" below is the canonical example of a named alternative.)

The manuscript is not the oracle. When the computed value disagrees with the manuscript, do not presume the code is correct and the paper stale — nor the reverse. A refactor may have broken a previously-correct table (the on-disk output is the buggy one), or the paper may carry an old number. The computed value is a challenger, not ground truth. Report a mismatch as "one of {paper, code} must change — isolate which," never "revert the code to match the paper." This prevents the trap of reverting a genuine bug-fix just to make the paper 'reproduce.'

Downgrade rule: FAIL → EXPLAINED

A FAIL may be downgraded to EXPLAINED only when a specific named alternative is recorded for that exact claim — in the passport entry's notes: field (passport mode) or the audit report's author-note column (default mode). Example of a valid note:

"reghdfe vs feols clustering-df adjustment; under the reghdfe small-sample correction the published value is −1.19, within rounding of the script's −1.187. CODE-CORRECTED pending."

The author is the auditor: the skill stages the two-sided comparison (reported value and computed value, both shown); the human writes the one-line named alternative; the skill records it and thereafter respects it. Tag the resolution PAPER-CORRECTED, CODE-CORRECTED, or DEFENSIBLE-ALTERNATIVE.

Hard floor — never downgradable to EXPLAINED:

• A blank note, "unclear", "looks fine", or any note that does not name a concrete alternative spec.
• An UNMATCHED claim (no computed counterpart to compare against).
• A flat numerical contradiction with no alternative offered.

(Citation/existence claims are out of scope here — /verify-claims owns those, and applies the same named-alternative softening on its side.)

Repeated EXPLAINED is a signal (two-strikes)

Reuse the two-strikes rule from review-paper --adversarial and summary-parity.md: if the same claim is downgraded to EXPLAINED in two consecutive audits without ever being corrected to PASS (the author keeps invoking the alternative but never updates paper or code), stop treating it as quietly resolved. Surface it prominently in Phase 5 — "this contested number has been EXPLAINED twice but never corrected" — so a standing disagreement can't hide behind a recorded note indefinitely. In passport mode, detect this by comparing the current status/notes against the prior audit's.

Phase 4c: Horizontal check — the displays must agree with each other

Phase 4 compared the claim to the code. This phase compares the claim's displays to each other, pairwise over the displays collected in Phase 2b:

1. Round both sides to the coarser precision. Two displays agree when they are equal after both are rounded to the smaller of their two display_precision values. 0.34 in a deck against 0.342 in the paper agrees at 2 decimals; 0.29 against 0.342 does not.
2. No tolerance applies. The tolerance: block governs the vertical comparison, where two measurements are compared. Two displays of one number are two copies — after the rounding in step 1 they either match or they do not.
3. Dispositions, in the vocabulary of Phase 4b:
   • PASS — every pair agrees.
   • FAIL — any pair disagrees, or any display is NOT_FOUND. Never downgradable to EXPLAINED: a named alternative spec explains why code and paper differ and has nothing to say about why two copies of one number differ. If a display genuinely shows a different specification, it is a different claim and belongs in its own passport entry.
   • UNCHECKED — the claim declares no appears_in list. Reported, non-blocking; the fix is to declare the displays.
4. Report which side moved, not which side is right. Name the artifact whose value differs from the rest, and when it was last touched; the author decides whether the deck is stale or the paper is. Do not hand-edit one display to match its sibling — re-derive both from the output the claim points at (replication-protocol.md → Anti-patterns).

Phase 5: Report

Write quality_reports/reproducibility_audit_[manuscript-name].md:

# Reproducibility Audit: [Manuscript Title]

**Date:** [YYYY-MM-DD]
**Manuscript:** [path]
**Outputs directory:** [path]
**Tolerance source:** .claude/rules/replication-protocol.md

## Summary

| Status | Count |
|---|---|
| PASS (vertical and horizontal) | N |
| FAIL (diff > tolerance, no named alternative) | M |
| EXPLAINED (out of tolerance, named alternative recorded) | E |
| UNMATCHED (manual review) | K |
| HORIZONTAL DRIFT (declared displays disagree, or a display not found) | H |
| HORIZONTALLY UNCHECKED (claim declares no `appears_in`) | U |
| **Overall verdict** | **PASS / FAIL** (FAIL iff M > 0 or H > 0; EXPLAINED does not fail the audit) |

## PASS (all within tolerance)
| Claim | Reported | Computed | Diff | Tolerance |
|---|---|---|---|---|
| Table2_col3_ATT | -1.632 (0.584) | -1.628 (0.591) | 0.004 / 0.007 | 0.01 / 0.05 |

## FAIL (outside tolerance — BLOCKER)
| Claim | Reported | Computed | Diff | Tolerance | Location in paper | Author note (name a concrete alternative to downgrade → EXPLAINED) |
|---|---|---|---|---|---|---|

## EXPLAINED (out of tolerance; defensible named alternative recorded — non-blocking, carry into response-to-referees)
| Claim | Reported | Computed | Named alternative (why the gap is defensible) | Resolution |
|---|---|---|---|---|
| Table3_col2_ATT | -1.187 | -1.19 | reghdfe vs feols clustering-df adjustment | DEFENSIBLE-ALTERNATIVE |

## UNMATCHED (manual review)
| Claim | Raw context | Candidate sources |
|---|---|---|

## HORIZONTAL DRIFT (one number, two values — BLOCKER)
| Claim | Display A | Display B | Compared at | Values | Which side moved |
|---|---|---|---|---|---|
| C3 | manuscript.tex — Table 1, Col 2 | Slides/Lecture04_Results.tex — frame 'Main result' | 2 decimals | 0.34 vs 0.29 | deck older than the claim's output_file |

## HORIZONTALLY UNCHECKED (no `appears_in` declared)
| Claim | Primary display | Other artifacts to declare |
|---|---|---|

## Environment
[sessionInfo excerpt]

## Next steps
1. Resolve each FAIL row — either correct the manuscript, rerun the analysis, or (if the gap is a defensible alternative spec) record a concrete named alternative to downgrade it to EXPLAINED.
2. Resolve each HORIZONTAL DRIFT row — regenerate the lagging display from the claim's output, never by retyping the other display's value. A `NOT_FOUND` display is fixed by correcting the `locator`, not by dropping the entry.
3. Review UNMATCHED rows — add explicit lookup keys or widen the search scope.
4. Declare `appears_in` for the HORIZONTALLY UNCHECKED rows — every artifact a reader sees the number in.
5. Review EXPLAINED rows before submission — each should map to a sentence in the response-to-referees.
6. After zero FAILs and zero HORIZONTAL DRIFT (EXPLAINED rows allowed), the paper is replication-ready.

Exit behavior

• All PASS (or PASS + EXPLAINED): exit 0, summary printed.
• Any HORIZONTAL DRIFT: exit 1, on the same footing as a FAIL — two artifacts showing different values for one number is a defect whichever of them is right. HORIZONTALLY UNCHECKED rows warn (exit 0) and are listed.
• Any FAIL: exit 1, summary printed to stderr. This makes the skill usable as a /commit pre-commit gate — see replication-protocol.md for the enforcement pattern. EXPLAINED rows do NOT count as FAIL and never trigger exit 1 — they are surfaced, not blocking. The gate keeps its full teeth for genuine FAILs (no named alternative) and for fabricated/UNMATCHED claims.
• UNMATCHED > 0 (with 0 FAIL): exit 0 with warning — user must manually review.

Source-language coverage

The skill compares manuscript claims against outputs in three source-language ecosystems:

Stata-specific notes (v1.9.0):

• .dta outputs are read via haven::read_dta() (R), pyreadstat.read_dta() (Python), or by parsing the corresponding esttab .tex if the table-cell value is what the manuscript cites.
• Manuscript cell \input{scripts/stata/_outputs/tab_main.tex} is the strongest provenance signal — the cell value comes mechanically from the .do file. Match the location in the .tex to the regression call in 03_analyze.do.
• Clustering df adjustments can differ between reghdfe and base reg, cluster(). If a SE mismatches at the 2nd decimal, the tolerance in replication-protocol.md covers it; if it mismatches at the 1st decimal, investigate the df adjustment.

Passport-mode (v1.9.0)

When quality_reports/passports/<paper-slug>.yaml exists, the skill operates in passport mode: instead of emitting a one-shot report, it reads, updates, and rewrites the passport file in place.

• For each claims: entry in the passport, perform the same numeric audit as the default mode (extract reported value from manuscript at location, locate computed value at source_file:source_line / output_file:output_field, compare against tolerance:), then the horizontal sweep of Phases 2b and 4c over the entry's appears_in list.
• After each claim is audited, update status in place:
  • PASS → claim within tolerance and every declared display agrees at the coarser precision.
  • FAIL → claim outside tolerance and the entry's notes does not name a concrete alternative; or two declared displays disagree; or an appears_in entry could not be located in its file. Record the discrepancy in notes — reported vs computed for a vertical FAIL, the two display values and their paths for a horizontal one. Blocks (exit 1).
  • EXPLAINED → claim outside tolerance but the entry's notes already records a specific named alternative spec (not blank, not "unclear"). The skill reads notes on its next run and resolves the same out-of-tolerance claim to EXPLAINED instead of FAIL — surfaced, non-blocking. The hard floor still applies: an UNMATCHED claim or a note without a named alternative stays FAIL.
  • STALE → if source_file, output_file, or any appears_in path has a modification time later than last_verified_on, mark STALE and re-run the audit logic (after the rerun, status becomes PASS / FAIL / EXPLAINED — STALE is transient).
• Update last_verified_on and last_verified_by: "/audit-reproducibility" per claim.
• Update paper.last_audit at the top level.

If a claim in the manuscript is detected that has no matching passport entry, emit an UNVERIFIED warning — the author should add it (passport scope is author-curated, not auto-populated, to avoid bad inferences).

Passport mode does NOT delete passport entries. If a claim disappears from the manuscript, the passport entry remains with a STALE status — the author decides whether to delete (claim retracted) or update the entry's location (claim moved).

Nor does it add appears_in entries on its own. A display the sweep happens to notice — the same value in a deck or a supplement that the passport never declared — is reported as a suggestion in the HORIZONTALLY UNCHECKED table; the author declares it. Same reasoning as the no-auto-populate rule above: an inferred display that is actually a different quantity would fail the horizontal check forever.

See .claude/rules/replication-protocol.md "Claims Provenance: passport.yaml" for the full schema and integration points (/commit, /review-paper).

Cross-references

• .claude/rules/replication-protocol.md — the tolerance contract + passport schema.
• templates/passport-template.yaml — starter file to copy for a new paper; the appears_in list is where displays are declared.
• .claude/hooks/claim-reconcile.py — the event-driven nudge: writing a tracked script, output, or declared display names the claims to re-audit. It counts declarations; this skill does the comparing.
• .claude/skills/review-r/SKILL.md — catches code-style issues; this skill catches NUMERICAL reproducibility.
• .claude/skills/diagnose/SKILL.md — when a claim resolves to FAIL and you need to localize which pipeline step produced the out-of-tolerance value, hand off to /diagnose (single-claim root-cause: reproduce → minimise → bisect).
• .claude/skills/review-paper/SKILL.md — content review; pair with this skill for a full pre-submission audit.
• .claude/skills/replication-package/SKILL.md — gates on this skill before assembling the AEA DCAS deposit.
• .claude/skills/capture-environment/SKILL.md · .claude/skills/disclosure-check/SKILL.md — environment capture + restricted-data screening downstream.

What this skill does NOT do

• Re-run your analysis. The skill compares CURRENT outputs against manuscript claims. If the outputs are stale, re-run your pipeline first (the pre-flight phase will warn).
• Catch wrong specifications. A regression that compiles cleanly and produces a reproducible -1.632 is reproducible. Whether -1.632 is the RIGHT estimand is a review-paper / domain-reviewer question.
• Check external package versions. The sessionInfo.txt capture lets a reviewer see the env; pinning versions is on the user (via renv.lock or a DESCRIPTION file).

Long batch reruns: use the Monitor tool (Apr 2026)

When /audit-reproducibility is asked to verify all numeric claims in a paper, the safest approach is to re-run the full pipeline (00_run_all.R or equivalent) and compare the regenerated outputs to the manuscript values. For pipelines that take more than a couple of minutes, background-launch the rerun and use Anthropic's Monitor tool (Apr 2026 Week 15) to stream stdout. The audit can react to errors mid-stream rather than waiting for the entire pipeline to finish before noticing a failed step.