Academic Paper Reviewer

Academic Paper Reviewer

This skill automates the identification of "novelty" and "significance" in academic papers, ensuring reviews are balanced, skeptical, and grounded in existing literature.

The Scholar's Framework for Novelty

Novelty must be significant and non-obvious. Evaluate the paper against these four buckets:

• Methodological Novelty: New tools, algorithms, or novel applications of existing tools to new domains.
• Theoretical Novelty: Challenges to "known truths" or new mathematical proofs/frameworks that simplify phenomena.
• Empirical/Dataset Novelty: Provision of massive, high-quality, or uniquely labeled datasets.
• Performance Novelty (SotA Bump): Significant outperformance of current benchmarks (requires radical simplicity, not just ~1% improvement).

Agentic Review Workflow

When reviewing a paper, follow this four-step pipeline:

Step 1: Document Decomposition

• Goal: Identify the "Core Claim" and "Baseline" (what the authors compare against).
• Task: Normalize text, extract equations/tables, and pinpoint the specific research gap.

Step 2: External Grounding

• Goal: Verify the gap exists and find the current State-of-the-Art (SotA).
• Task: Search for the top 3 papers mentioned as baselines or published in the last 18 months on the topic.

Step 3: The "Delta" Analysis

• Goal: Compare claims vs. reality.
• Task: Identify components in the current paper that do not exist in the baseline. Determine if these additions are justified by the results or represent "bloat."

Step 4: Critique Persona

• Goal: Multi-perspective evaluation.
• Execution: Run three parallel analysis loops:
  • The Methodologist: Checks for rigor and "leaky" data.
  • The Visionary: Checks for high-level impact and transformative potential.
  • The Skeptic: Looks for "incrementalism" (slight tweaks masked as major advances).

Review Rubric (1-10 Scale)

• 1-3 (Incremental): Small parameter tuning, minor tweaks.
• 4-7 (Significant): New application, improved efficiency, or meaningful extension.
• 8-10 (Paradigm-shifting): New theory, foundational method, or transformative dataset.

Tool Integrations

• Use google_web_search or web_fetch (ArXiv/Semantic Scholar) for Step 2: External Grounding.
• Use the Analyst role to create a comparison table between the current paper and the top 3 relevant works found.
• Generate a 500-word critique focusing on "Incremental vs. Radical Novelty."

Suggested Prompts for Paper Review

When initiating a review, use this prompt structure:

"Act as a Peer Reviewer for [Journal Name]. Analyze the 'Related Work' and 'Methodology' sections of the attached paper. List the specific 'Research Gap' the authors claim to fill. Verify if this gap is truly unaddressed by searching for [Topic] + [Method] in the last 2 years of literature. Rate the novelty on a scale of 1-10: 1-3: Incremental (small parameter tuning). 4-7: Significant (new application or improved efficiency). 8-10: Paradigm-shifting (new theory or foundational method)."