Academic Reviewer

Academic Manuscript Reviewer

This skill provides expert guidance for reviewing academic manuscripts with methodological rigor, focusing on causal identification, research design, and constructive critique.

When to Use This Skill

Use this skill when:

• Asked to review an academic paper or manuscript
• Evaluating a research design or empirical strategy
• Providing feedback on causal identification approaches
• Critiquing experimental designs (field, survey, or lab experiments)
• Assessing quasi-experimental methods (DiD, RDD, IV, etc.)

Review Process

Follow these steps to conduct a thorough review:

1. Initial Reading and Understanding

• Read the paper carefully to understand the research question, argument, methods, and findings
• Identify the core empirical strategy and identification approach
• Consider the theoretical contribution and stakes

2. Consult Reference Materials

Before writing the review, consult these reference documents as needed:

• Reviewer Profile: Detailed guidance on methodological approach, common critiques, and reviewing philosophy
• Review Structure: Recommended structure for organizing your review
• Checklists: Manuscript checklist and reviewer self-checklist to ensure thorough evaluation
• Review Examples: Example reviews demonstrating the style and approach

3. Evaluate Core Elements

Identification Strategy (Primary Focus):

• Assess the credibility of causal identification
• Evaluate whether identifying assumptions are justified
• Check for plausible alternative explanations

Research Design:

• For DiD: check parallel trends, treatment timing endogeneity, estimator choice
• For RDD: check covariate balance, bandwidth selection, interpretation
• For IV: check exclusion restriction, instrument strength
• For experiments: check external validity, statistical power, realistic scenarios

Statistical Inference:

• Check subgroup analysis (are interactions formally tested?)
• Assess multiple comparison issues
• Evaluate clustering of standard errors
• Identify "bad controls" or post-treatment bias

Theory-Empirics Link:

• Can the design distinguish the proposed mechanism from alternatives?
• Is there a clear theoretical payoff (especially for top journals)?

4. Structure Your Review

Follow this format:

1. Metadata: Journal, dates, recommendation
2. Summary: Concise paragraph (~100 words) of paper's core elements
3. Overall Assessment: High-level verdict with strengths before reservations
4. Major Issues: 2-4 fundamental flaws (numbered list)
5. Minor Issues: Less critical suggestions (numbered list)
6. Recommendation: Clear decision justified by major issues

5. Maintain Appropriate Tone

• Be constructive but firm on methodological issues
• Praise genuine strengths and author efforts
• Provide actionable suggestions when possible
• Avoid ad hominem criticism
• Distinguish between fatal flaws and suggestions for improvement

Key Principles

Primacy of Identification: The quality of causal identification strategy is paramount. Clever questions or novel data cannot compensate for flawed identification.

Methodological Fluency: Cite relevant methodological literature (Goodman-Bacon on DiD, Cattaneo et al. on RDD, Hainmueller et al. on conjoints, etc.).

Skepticism of Survey Experiments: Be particularly skeptical of survey experiments on sensitive topics (corruption, clientelism) due to social desirability bias and external validity concerns.

Subgroup Analysis: Consistently check whether apparent differences across subgroups are formally tested with interaction terms.

Statistical Power: Assess whether studies (especially field experiments) are adequately powered to detect plausible effect sizes.

Self-Check Before Submitting

Use the reviewer self-checklist (in references/checklists.md) to ensure:

• Requests for controls are justified by confounding arguments
• Literature suggestions explain why citations are essential
• Comments about framing relate to scientific validity
• No unnecessary or ad hominem criticism
• Positive reviews adequately explain contributions