Research Paper Review

Research Paper Review

Assist researchers in reviewing, analyzing, and criticizing academic papers systematically and thoroughly.

When to Use

• User asks to review, summarize, or critique a research paper
• User shares a PDF or link to an academic paper
• User wants to assess methodology, contributions, or novelty
• User needs help writing a peer review
• User wants to compare multiple papers or do a literature survey
• User wants to improve a paper before submission (pre-submission review)
• User wants to check for numerical/statistical inconsistencies
• User wants venue-specific feedback (conference, journal, or preprint)

Inputs

• Paper content: PDF, LaTeX source, or plaint text
• Target Venue (optional but recommended): conference, journal, or preprint target - Example: "Models 2026", "TOSEM", "Sosym", "arXiv preprint"
• Type of paper (important — ask the user if not provided): the review criteria change substantially by type. Common types: "full research" paper, "short" paper, "new ideas"/vision paper, "tool demo" paper, "poster", "extended abstract"/"short abstract", "journal"/"extended" paper. If the user does not state the type, ask before reviewing (or state the assumed type explicitly in the review).
• Explicit reviewing guidelines (optional): if available provide a description or URL with the reviewing criteria.

Review Workflow

###Step 0: Pre-Processing / Venue Context

• If target venue and/or the type of paper are provided, include them as context for all subsequent steps:

“Review this paper as if it is intended for [TARGET VENUE] as a [TYPE PAPER] submission. Consider typical standards, expectations, page limits, scope, and audience for this venue and type of paper”

• Calibrate expectations to the paper type. Do not apply full-paper standards to every submission. In particular, do not fault a paper for missing something its type does not require (e.g., do not criticize a tool demo paper for lacking a real-world validation study, or a vision paper for lacking large-scale experiments). Use the table below as the baseline, then adjust with the venue's own guidelines:

• If the paper type is unknown, ask the user (or state the assumption you are reviewing under) before applying validation criteria.

• Optional: Use the provided reviewing guidelines or try to find them on the venue website (if available) with standards for methodology, novelty, empirical rigor, validation and formatting. Venue-specific guidelines override the defaults above.

Step 1: Read the Paper

Identify the format and read accordingly:

• PDF: Use the Read tool with the pages parameter for large documents (max 20 pages per request).
• LaTeX source: Read the main .tex file first. Look for \input{} or \include{} commands to find additional sections, figures, and bibliography files. Use Grep to search for key commands like \begin{abstract}, \section, \cite across all .tex files.
• Multiple files: Use Glob with **/*.tex to find all source files, then read them in logical order (main file → sections → appendix).

In all cases, skim the abstract, introduction, and conclusion first to get the big picture before diving into details.

Step 2: Structured Summary

Show that you understand the paper by producing a summary covering:

1. Problem Statement — What problem does the paper address? Why does it matter?
2. Contributions — What are the claimed contributions? (list them)
3. Approach/Methodology — How do the authors solve the problem?
4. Key Results — What are the main findings/metrics?
5. Limitations — What are the acknowledged (and unacknowledged) limitations?

Step 3: Numerical & Consistency Checks - most info from linkedin

This is where LLM-assisted review adds the most value, catching things humans easily miss during manual review. Run these checks systematically:

• Numbers across text, tables, and figures: Do values reported in the text match what's in the tables? Do figures reflect the data described?
• Statistical consistency: Do p-values, confidence intervals, and effect sizes align? Are sample sizes consistent throughout?
• Calculations: Verify percentages, averages, sums. Check that reported improvements (e.g., "30% improvement") match the actual numbers.
• Internal references: Do all \ref, \cite, figure/table references resolve? Are there dangling references or wrong numbering?
• Acronyms: Are all acronyms defined on first use?
• Terminology consistency: Is the same concept always referred to with the same term?
• Citations: Do all citations exist? Is citation style uniform (i.e. all conference papers are cited using the same fields, same for other venues)

Even minor errors (typos, broken references, wrong numbering) matter. reviewers often use these as signals that the paper was not carefully prepared.

Step 4: Critical Analysis

Evaluate the paper on these dimensions, weighting them according to the paper type (see the calibration table in Step 0). For example, weight Reproducibility and Soundness heavily for a full/journal paper, but weight Significance (tool usefulness, availability, demo clarity) more than exhaustive Soundness for a tool demo, and Novelty/Clarity over Soundness for a vision paper. When a dimension is not applicable to the paper type, say so explicitly rather than penalizing the paper for it.

Step 5: Provide Actionable Feedback

Structure feedback as:

• Strengths — What the paper does well (be specific, cite sections)
• Weaknesses — What could be improved (be constructive, suggest fixes)
• Questions for Authors — Things that need clarification
• Minor Issues — Typos, formatting, citation issues, broken references
• Venue-Specific Recommendations — Highlight alignment issues, potential improvements to meet venue expectations

Step 6: Start here

Write down a list of the top 10 most immediate actions that the author should address.

These should be the ones that will bring the best "bang for the buck", i.e. actions that generate the most benefit relative to the cost of implementing them.

Output Format

Use this template for the review:

# Paper Review: [Title]

## Summary
[2-3 paragraph summary]

## Strengths
- S1: ...
- S2: ...

## Weaknesses
- W1: ...
- W2: ...

## Questions for Authors
- Q1: ...

## Minor Issues
- ...

## Venue-Specific Recommendations
- V1: ...
- V2: ...

## Overall Assessment
[1 paragraph verdict: accept/revise/reject with justification]

## Top actions . Start here
- T1: ...
- T2: ...


## Confidence
[Your confidence level in this review: low/medium/high, and why]