Claims Audit

Claims Audit

Verify that every number, every figure, and every reported experiment in a manuscript is traceable, reproducible, and complete. This skill subsumes negative-results reporting and figure integrity into a unified audit.

The Iron Law:

EVERY NUMBER IN THE MANUSCRIPT MUST TRACE TO A SPECIFIC FILE. NO EXCEPTIONS.

A number without a source file is not a finding — it is an assertion. Assertions do not belong in scientific papers.

When to Use

Use this skill:

• After any manuscript draft is produced
• Before submitting to any journal
• Before any internal review round
• When verification-before-publication requests a claims audit
• When a coauthor asks "where does this number come from?"

Do NOT use this skill to:

• Run experiments (use experiment-design)
• Generate figures (use the analysis/visualization scripts)
• Write manuscript sections (use the manuscript-writer agent)

Announce at Start

At the beginning of every session using this skill, state:

"I'm running the claims-audit skill. This will audit number traceability, figure integrity, and completeness of reported results."

The Three Audit Components

Component A: Number Traceability

For every quantitative claim in the manuscript — correlation coefficients, p-values, sample sizes, percentages, improvement margins, AUC values, effect sizes, anything with a number — the claim must be traceable to a specific file and a specific line or cell that produced it.

Process:

1. Extract every quantitative claim from the manuscript. Work section by section: Abstract, Introduction (any quantitative statements), Methods (N, parameters), Results (all reported statistics), Discussion (any numbers not already extracted).
2. For each claim, identify the source file in results/ (or equivalent output directory).
3. Find the exact line, JSON key, CSV row, or notebook cell that produces that number.
4. Compare the manuscript value to the source value character by character. Rounding is acceptable only if it is the correct rounding of the source value — verify this explicitly.
5. Flag any discrepancy as CRITICAL.

What counts as a quantitative claim:

Output table — Number Traceability:

| Claim | Manuscript Location | Source File | Source Line | Match? | Status |
|---|---|---|---|---|---|
| r = 0.86 | Results §3.1, L42 | results/cv_metrics.json | key: mean_pearson_r | YES | OK |
| p < 0.001 | Results §3.2, L55 | results/group_stats.csv | row 22, col p_value | NO (actual: 0.003) | CRITICAL |
| N = 47 | Methods §2.3, L18 | data/processed/cohort_subjects.csv | row count | YES | OK |
| "23% improvement" | Abstract, L6 | results/comparison_table.csv | (no matching value) | MISSING | CRITICAL |

Discrepancy severity:

Component B: Figure Integrity

Every figure in the manuscript must be generated by a version-controlled script. A figure that was manually created, manually edited in image software after script generation, or that has no traceable generating script is a CRITICAL finding.

Process:

1. List every figure in the manuscript (Fig 1, Fig 2, Fig 3, supplemental figures, etc.).
2. For each figure, identify the generating script. Check: scripts/, src/, figures/, notebooks/, or any path referenced in a comment at the top of the figure file.
3. Verify the script file exists.
4. Verify the script is runnable: required imports exist, data dependencies are present, no hardcoded paths to files that no longer exist.
5. Check for signs of manual post-processing: mismatched metadata (file modified after script last ran), vector graphics with manually moved elements, rasterized layers on top of programmatic output.
6. Any manually edited figure is CRITICAL. The fix is to regenerate the figure programmatically and update the script to produce the desired appearance.

What "manually edited" means:

• Opening the figure file in Illustrator, Inkscape, or Photoshop and saving it
• Modifying axis labels, tick marks, or annotations that should come from the script
• Adjusting colors, line widths, or fonts outside the script
• Adding arrows, boxes, or callouts in image software that should be in the plotting code
• Cropping or resizing in a way not reproducible by re-running the script

What is NOT a manual edit:

• Exporting at a different DPI from the same script (document this)
• Assembling multi-panel figures with a defined layout script
• Converting between formats (SVG to PDF) via command-line tools documented in the build process

Output table — Figure Integrity:

| Figure | Script | Exists? | Runnable? | Manual Edit? | Status |
|---|---|---|---|---|---|
| Fig 1 | scripts/plot_fig1_overview.py | YES | YES | NO | OK |
| Fig 2 | scripts/plot_fig2_scatter.py | YES | YES | NO | OK |
| Fig 3 | — | NO | — | — | CRITICAL: no generating script |
| Fig 4 | scripts/plot_fig4_heatmap.py | YES | YES | YES (Illustrator metadata) | CRITICAL: manual edits detected |
| Supp Fig 1 | notebooks/supp_analysis.ipynb cell 14 | YES | YES | NO | OK |

Runnable check — minimum requirements:

• Script file exists at the stated path
• All imports resolve (no missing packages)
• Input data files referenced in the script exist
• Script can be executed end-to-end without errors (run a dry-run check or verify with the experiment log)
• Output figure file path in script matches the actual figure file used in the manuscript

Component C: Completeness (Negative Results)

Every experiment that was run must be reported in the manuscript, or explicitly accounted for. Omitting a non-significant result is not "keeping the paper focused" — it is publication bias.

Process:

1. List every experiment that was run. Sources: the experiment log (docs/eureka/records/experiment-log.md or equivalent), the results/ directory structure, config files in configs/, git commit history with experiment tags.
2. List every result reported in the manuscript (main results, supplemental, appendix).
3. Cross-reference: every run experiment must appear somewhere in the manuscript, or in a documented decision not to report it with a stated reason.
4. Flag any experiment that was run but not reported as CRITICAL if the result was non-significant. Flag as WARNING if the result was significant but unreported (possible selective reporting of direction or magnitude).

What "reported" means:

An experiment is reported if:

• Its primary metric appears in the Results section or Supplementary Material, OR
• The manuscript explicitly acknowledges the experiment was run and states the result (including null), OR
• The experiment was superseded by a later version with an explicit note in the methods

An experiment is NOT reported if:

• It was run and produced a result, and the manuscript says nothing about it
• The paper says "we ran X" but does not state the result of X
• The result was non-significant and the paper implies the approach was never tried

Output table — Completeness:

| Experiment | Run? | Reported? | Significant? | Status |
|---|---|---|---|---|
| Baseline method A | YES | YES | YES (r = 0.72) | OK |
| Baseline method B | YES | YES | YES (r = 0.79) | OK |
| Proposed method | YES | YES | YES (r = 0.86) | OK |
| Ablation: feature set X removed | YES | NO | NO (r = 0.61) | CRITICAL: unreported negative result |
| Ablation: feature set Y removed | YES | YES | NO (r = 0.58) | OK (reported in Supp Table 2) |
| Sensitivity: alternate preprocessing | YES | NO | NO (p = 0.18) | CRITICAL: unreported null result |
| Sensitivity: alternate subgroup | YES | NO | YES (r = 0.84) | WARNING: unreported significant variant |

Non-significant does not mean unreportable. A non-significant ablation result is direct evidence for the contribution of the ablated component. Omitting it makes the paper's contribution claim unverifiable.

The Audit Process

Run these steps in order. Do not skip steps because a previous audit was performed. Manuscripts change.

Step 0: Dispatch the traceability-auditor computational subagent (NEW in v1.10.0)

Before manual inspection, dispatch the subagent at skills/claims-audit/traceability-auditor-prompt.md via the Task tool.

Fill placeholders:

• {MANUSCRIPT_PATH} → path to the manuscript source (paper/main.tex, manuscript.md, or equivalent)
• {RESULTS_DIR} → path to the results directory (typically results/)
• {TOLERANCE} → 0.01 for floats, 0 for integers (adjust if project uses unusual precision)

The subagent extracts every numeric claim via regex, scans the results directory, and returns a machine-readable table of matches/approximate/untraceable. This eliminates the mechanical cross-referencing labor — the main agent then applies judgment to the exceptions (Must-fix untraceable values, Should-fix approximate matches).

Iron Law preserved: the subagent automates extraction, not judgment. The main agent must still investigate every untraceable value: is the number a typo, a source-file gap, or a stale carry-over from an earlier manuscript version?

If the subagent returns Status: Approved (Must-fix count = 0), proceed to Steps 1-7 with the verified list. If it returns Status: Issues Found, resolve the Must-fix items first, then proceed.

Step 1: Extract all quantitative claims (verify subagent output)

The traceability-auditor has produced a table of claims. Cross-check: did it miss any manuscript numbers? Domain-specific formats (e.g., chemical structure values, engineering units) may not match the default regex set. Add any missed claims to the table manually.

Step 2: Trace each claim to a source file

For claims the subagent marked as untraceable, find the source file manually. Check: notebooks/ output cells, logs/, inline in scripts (not acceptable — this is a flag). Record the source file path and the exact location within the file.

Step 3: Verify each number matches

For claims the subagent marked as approximate, investigate: rounding or actual discrepancy? Any discrepancy beyond documented rounding is CRITICAL.

Step 4: Extract all figures

List every figure in the manuscript. For each, determine the generating script.

Step 5: Verify each figure is script-generated

Apply the runnable check. Check for manual edits. Record findings.

Step 6: List all experiments run

Retrieve the experiment log and scan the results directory. List every experiment.

Step 7: Cross-reference with manuscript

Compare the run experiments list against reported results. Flag unreported experiments.

Step 8: Generate the audit report

Compile the three output tables and the checklists. Write the audit report. Save it.

Verification Checklist

After completing all three components, confirm:

Number Traceability

• Every quantitative claim in Abstract has a source file
• Every quantitative claim in Results has a source file
• Every quantitative claim in Methods (N, parameters) has a source file
• Every quantitative claim in Discussion has a source file
• No CRITICAL mismatches exist
• All source files are in version control

Figure Integrity

• Every figure has a named generating script
• Every generating script exists at its stated path
• Every generating script is runnable end-to-end
• No figure has been manually post-processed
• All generating scripts are in version control

Completeness

• Experiment log is current (all recent runs are recorded)
• Every run experiment is accounted for in the manuscript
• No non-significant result is silently omitted
• Supplementary material is checked, not just main text

Common Rationalizations

Red Flags

Stop the audit and flag immediately if:

• A number in the manuscript cannot be found anywhere in the results directory — the number may have been invented or carried over from a previous version of the study
• A figure file's modification timestamp is more recent than the last run of its generating script — the figure may have been edited after generation
• The experiment log has fewer entries than the number of result files in the results directory — experiments were run without logging
• Any claim uses the word "approximately" without citing the exact value it approximates
• Two different sections of the manuscript cite the same statistic with different values
• The results directory contains subdirectories with names like old/, backup/, pre_fix/ that contain different values than the current results — the manuscript may reference stale numbers
• A figure appears in the manuscript but the corresponding file in figures/ does not exist (figure embedded from memory or another project)

Output: Audit Report

Save the completed audit report to:

docs/eureka/audits/YYYY-MM-DD-claims-audit.md

Where YYYY-MM-DD is today's date.

Audit Report Template

# Claims Audit Report
**Date:** YYYY-MM-DD
**Manuscript version:** [git commit hash or version label]
**Auditor:** [name or session ID]
**Overall status:** PASS / FAIL (FAIL if any CRITICAL items remain unresolved)

---

## Component A: Number Traceability

| Claim | Manuscript Location | Source File | Source Line | Match? | Status |
|---|---|---|---|---|---|
| [fill in] | | | | | |

**Traceability summary:**
- Total claims audited: N
- OK: N
- CRITICAL: N
- Unresolved CRITICAL items: [list]

---

## Component B: Figure Integrity

| Figure | Script | Exists? | Runnable? | Manual Edit? | Status |
|---|---|---|---|---|---|
| [fill in] | | | | | |

**Figure integrity summary:**
- Total figures audited: N
- OK: N
- CRITICAL: N
- Unresolved CRITICAL items: [list]

---

## Component C: Completeness

| Experiment | Run? | Reported? | Significant? | Status |
|---|---|---|---|---|
| [fill in] | | | | | |

**Completeness summary:**
- Total experiments run: N
- Fully reported: N
- Unreported (CRITICAL): N
- Unreported (WARNING): N
- Unresolved CRITICAL items: [list]

---

## Verification Checklist

### Number Traceability
- [ ] Every quantitative claim in Abstract has a source file
- [ ] Every quantitative claim in Results has a source file
- [ ] Every quantitative claim in Methods has a source file
- [ ] Every quantitative claim in Discussion has a source file
- [ ] No CRITICAL mismatches exist
- [ ] All source files are in version control

### Figure Integrity
- [ ] Every figure has a named generating script
- [ ] Every generating script exists and is runnable
- [ ] No figure has been manually post-processed
- [ ] All generating scripts are in version control

### Completeness
- [ ] Experiment log is current
- [ ] Every run experiment is accounted for
- [ ] No non-significant result is silently omitted
- [ ] Supplementary material checked

---

## Red Flags Encountered

[List any red flags found, or "None."]

---

## Required Actions Before Submission

[List all CRITICAL items and the action required to resolve each, or "None — audit passed."]

Integration

Called by: verification-before-publication — claims-audit is a required component of the pre-submission verification workflow. verification-before-publication will not return PASS if claims-audit has unresolved CRITICAL items.

Feeds into: submission-readiness — the audit report is cited as evidence in the submission readiness check.

Workflow position:

manuscript draft → claims-audit → [resolve CRITICALs] → verification-before-publication → submission-readiness

If claims-audit finds CRITICAL items, do not proceed to verification-before-publication. Resolve the CRITICALs first, then re-run the audit.

Skill Type

RIGID — Follow exactly. Every step is mandatory. Do not skip Component C because "we know we reported everything." Do not skip the source-file lookup for numbers you believe are correct.

The Iron Law exists because numbers drift between analysis and manuscript, figures get touched up, and convenient omissions happen gradually and with good intentions. The audit is not a sign of distrust — it is the verification step that makes the paper's claims defensible.

An audit that finds zero issues on first run should be treated with suspicion. Re-read the manuscript more carefully.