Paper Audit

Paper Audit

Claims in the paper must match code in the repository.

This skill audits research papers against their publicly available codebases to detect mismatches, missing implementations, and reproducibility issues. Inspired by Feynman's /audit workflow.

"Trust but verify." — Russian proverb

1. When to Audit

Audit when:

• Reviewing a paper with an accompanying GitHub repository
• Planning to build on someone's method (verify it works first)
• Reviewing for a conference/journal (check reproducibility)
• Teaching (show students what to look for)

Do NOT audit when:

• No code is available (audit becomes impossible)
• Paper is purely theoretical (no implementation to check)
• You are the author (use internal-critique instead)

2. The Audit Protocol

Step 1 — Paper Claim Extraction

Extract all verifiable claims from the paper:

Claim types:

• Hyperparameter: Learning rate, batch size, weight decay, architecture details
• Architecture: Layer counts, dimensions, activation functions, connectivity
• Data: Dataset version, preprocessing steps, augmentation pipeline
• Metric: Reported numbers with confidence intervals
• Ablations: Performance differences when components are removed

Step 2 — Code Mapping

For each verifiable claim, locate the corresponding code:

Paper Claim → Code Location → Verification Result

Example:
C1: "Adam optimizer, lr=0.001"
  → Code: train.py:45: optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
  → Result: ✅ MATCH

C2: "Batch size is 256"
  → Code: config.yaml: batch_size: 128
  → Result: ❌ MISMATCH (paper says 256, code uses 128)

C3: "95.2% accuracy on ImageNet"
  → Code: README.md reports 94.8%
  → Result: ⚠️ DISCREPANCY (0.4% gap, possibly different eval protocol)

Where to look:

Step 3 — Mismatch Classification

Step 4 — Reproducibility Check

Attempt to run the code (if environment permits):

# 1. Environment setup
pip install -r requirements.txt  # Check: are all dependencies listed?

# 2. Data availability
# Check: is data downloadable? Is preprocessing script provided?

# 3. Run training
python train.py --config config.yaml  # Check: does it run without errors?

# 4. Run evaluation
python eval.py --checkpoint model.pth  # Check: can you reproduce reported numbers?

Reproducibility score:

3. Audit Report Template

## Paper Audit: [Paper Title] by [Authors]

**Paper:** [arXiv/DOI link]
**Code:** [GitHub/repo link]
**Audit Date:** YYYY-MM-DD
**Auditor:** [Name/Agent]

### Claim Verification Matrix

| ID | Claim | Code Location | Result | Notes |
|---|---|---|---|---|
| C1 | Adam, lr=0.001 | train.py:45 | ✅ Match | Exact match |
| C2 | Batch size 256 | config.yaml:3 | ❌ Mismatch | Code uses 128 |
| C3 | 95.2% ImageNet | README.md | ⚠️ Discrepancy | README reports 94.8% |

### Mismatches Found

#### 🔴 Critical (0 found)
[None / list]

#### 🟡 Major (1 found)
1. **Different batch size** (C2)
   - Paper: 256
   - Code: 128
   - Impact: Could affect convergence and final performance
   - Recommendation: Clarify which was used for reported results

#### 🟢 Minor (1 found)
1. **README outdated** (C3)
   - README reports 94.8% but paper claims 95.2%
   - Possibly different evaluation protocol or checkpoint

### Reproducibility Score: 3/5

**What worked:**
- Code structure is clean and modular
- Dependencies are mostly listed

**Blockers:**
- Missing preprocessing script for custom dataset
- Hardcoded paths in config files

**Time to reproduce:** Estimated 4-6 hours (with fixes)

### Recommendations

1. **For authors:** Update README with correct hyperparameters and results
2. **For users:** Use batch_size=128 for fair comparison, or ask authors which was used
3. **For reviewers:** Flag batch size discrepancy in review

### Verdict

- [ ] **Reliable** — Claims match code, reproducible
- [ ] **Mostly reliable** — Minor discrepancies, reproducible with fixes
- [x] **Use with caution** — Major discrepancies, needs clarification
- [ ] **Not reproducible** — Critical mismatches or missing code

4. Common Audit Patterns

Pattern 1: The "Cherry-picked" Result

Symptom: Paper reports best run, code shows multiple runs with variance.

Detection:

# In code: results/ folder has multiple runs
# run_1: 94.1%, run_2: 94.8%, run_3: 95.2%, run_4: 94.5%
# Paper reports: 95.2% (best run, not average)

Verdict: Not necessarily dishonest, but should report mean ± std.

Pattern 2: The "Missing Ablation"

Symptom: Ablation table in paper, no ablation code in repo.

Detection:

# Search for ablation scripts
find . -name "*ablation*" -o -name "*ablate*"
# No results found

Verdict: Major concern. How were ablation numbers computed?

Pattern 3: The "Different Method"

Symptom: Paper describes complex method, code uses simple baseline.

Detection:

# Paper: "We propose a novel hierarchical attention mechanism..."
# Code: model.py uses nn.MultiheadAttention (standard PyTorch)

Verdict: Critical mismatch. Method is not implemented as described.

Pattern 4: The "Undocumented Preprocessing"

Symptom: Results don't match, preprocessing steps differ.

Detection:

# Paper: "We normalize images to [0,1]"
# Code: transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

Verdict: Major concern. Different preprocessing → different results.

5. Tools for Auditing

Manual Audit

• Read paper claims → Find code → Compare
• Best for: Deep understanding, thorough verification
• Time: 2-4 hours per paper

Semi-Automated Audit

• Use grep / ripgrep to find claim keywords in code
• Use diff to compare configs
• Best for: Quick checks, large codebases
• Time: 30-60 minutes per paper

Automated Audit (future)

• Parse paper PDF for claims (NLP extraction)
• Parse code for hyperparameters and architecture
• Auto-compare and flag mismatches
• Best for: Batch auditing, systematic reviews
• Time: 5-10 minutes per paper (after setup)

6. Integration with Other Skills

References

• Feynman /audit — Paper vs codebase mismatch audit
• Papers With Code — Database of papers and their code
• ML Reproducibility Challenge — Community effort to reproduce ML papers