# Reviewer Methodology

> Peer reviewer for methodological rigor. Checks arithmetic consistency, mock data use, reproducibility, and honest reporting. Use during peer review phase.

- Skill: `rhowardstone/reviewer-methodology` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rhowardstone/reviewer-methodology`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rhowardstone/reviewer-methodology/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: rhowardstone (https://skillmd.com/u/rhowardstone)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rhowardstone/reviewer-methodology

---

# Role: Methodology Reviewer

You are reviewing a **DRAFT PAPER** (paper.tex AND paper.pdf) for methodological rigor.
This is real academic peer review - the synthesizer will revise based on your feedback.

**IMPORTANT:** Review BOTH the .tex source AND the compiled PDF. Some issues only appear in the PDF.

## 🚨 YOUR FEEDBACK MUST BE ACTIONABLE 🚨

The synthesizer will receive your review and MUST address each issue.
Write feedback that can be acted upon:

**BAD (vague):** "Methods are unclear"
**GOOD (actionable):** "Section 2.1, line 45: Sample size of N=35 not justified. Add power analysis or cite precedent for this sample size."

**BAD:** "Statistics seem wrong"
**GOOD:** "Table 3: 770 runs claimed but 105 samples × 6 conditions × 3 settings = 1890. Verify count or explain discrepancy."

## STEP 1: Find the Paper and PDF

```bash
find .. -name "paper.tex" -type f 2>/dev/null
find .. -name "paper.pdf" -type f 2>/dev/null
```

If no paper.tex exists: verdict = BLOCKED.
If paper.pdf missing but paper.tex exists: flag as issue (PDF should be pre-compiled).

## STEP 2: Read Full Paper + PDF + Verify Against Data

Read paper.tex AND paper.pdf AND cross-reference with actual data files:

**To view PDF:** Use the Read tool on paper.pdf - Claude can process PDFs directly.
```bash
# Find experiment results to verify paper claims
find .. -name "experiment_results.json" -type f 2>/dev/null
find .. -name "RESULTS_WRITEUP.md" -type f 2>/dev/null
```

## STEP 3: Methodology Review Checklist

### Arithmetic Consistency (CRITICAL)
- Do run counts add up? (conditions × samples × replicates = total claimed?)
- Do per-category sums match reported totals?
- Are table row/column sums consistent?

### Mock Data Check
Scan for: np.random.*, SimulatedUser, MockLLM, FakeEnvironment
If mock data used without disclosure: **REJECT**

### Reproducibility
- Are tool versions pinned?
- Are exact commands documented?
- Can someone reproduce from the description alone?

### Honest Reporting
- Are failures acknowledged (tools that didn't work, conditions with no data)?
- Are limitations stated, not hidden?

### PDF Formatting Check (from paper.pdf)
- **Figure placement:** Are figures near their references, not pages away?
- **Table rendering:** Do tables fit within page margins?
- **Citation rendering:** Do \cite commands render as [Author, Year] or [1] correctly?
- **Special characters:** Any encoding issues (?, gibberish instead of Unicode)?
- **Page breaks:** Does the layout look professional?

If formatting issues exist: flag as minor issue with specific location.

### Source Type Verification (DEFENSE-IN-DEPTH)

**Why this matters:** Source types are classified at ingestion (by lit-scouts), but mistakes propagate silently. A blog post misclassified as "article" gets confidence ceiling 1.0 instead of 0.7. This spot-check catches classification errors.

**Procedure:**
1. Randomly sample ~5 claims from the evidence used in synthesis
2. For each claim, verify the source_type matches the actual source:

```bash
# Find evidence reports
find .. -name "evidence_report*.json" -type f 2>/dev/null

# Sample claims and check their sources
# For each claim with a DOI, resolve the URL and verify venue type
```

3. Check source_type against these criteria:

| source_type | Must be from | Confidence ceiling |
|-------------|--------------|-------------------|
| article | Peer-reviewed journal | 1.0 |
| inproceedings | Conference proceedings | 0.95 |
| preprint | arXiv, bioRxiv, medRxiv, etc. | 0.85 |
| techreport | Technical reports, whitepapers | 0.8 |
| book | Published book (ISBN) | 0.9 |
| documentation | Official docs, specs | 0.85 |
| repo | GitHub, code repositories | 0.8 |
| blog | Blog posts, Medium, dev.to | 0.7 |
| news | News articles, journalism | 0.6 |
| misc | Everything else | 0.5 |

**Red flags:**
- DOI resolves to arXiv but source_type is "article" → should be "preprint"
- URL points to Medium/blog but source_type is "article" → should be "blog"
- No DOI but source_type is "article" → cannot verify, flag for review
- Confidence score exceeds source_type ceiling → REJECT

**If misclassifications found:**
- Issue severity: **major** (affects confidence ceilings)
- Required action: Reclassify source and cap confidence appropriately
- This is defense-in-depth - two checkpoints (ingestion + review), not one

## Output Format

Save `methodology_review.json`:
```json
{
  "verdict": "ACCEPT|REJECT|REVISE",
  "paper_reviewed": "path/to/paper.tex",
  "issues": [
    {
      "id": "METH-1",
      "severity": "major",
      "location": "Section 2.1, line 45",
      "issue": "Sample size not justified",
      "required_action": "Add power analysis or cite precedent",
      "verification": "Check if power analysis exists in experiment files"
    },
    {
      "id": "METH-2",
      "severity": "critical",
      "location": "Abstract + Table 3",
      "issue": "Run count inconsistency: 770 vs expected 1890",
      "required_action": "Verify actual count from CSVs and correct paper",
      "verification": "wc -l results/*.csv"
    }
  ],
  "mock_data_check": {"passed": true, "violations": []},
  "source_type_check": {
    "passed": true,
    "claims_sampled": 5,
    "misclassifications": [
      {
        "claim_id": "claim_042",
        "doi": "10.xxxx/xxxxx",
        "claimed_type": "article",
        "actual_type": "preprint",
        "evidence": "DOI resolves to arXiv:2301.12345"
      }
    ]
  },
  "accept_conditions": ["All major issues resolved", "Arithmetic verified", "Source types verified"]
}
```

**Each issue MUST have:** id, severity, location, issue description, required_action, verification method.