108 Understanding 58ecad55

Reviewer Expectations by Venue

Understanding what reviewers look for at different venues is essential for crafting successful submissions. This guide covers evaluation criteria, common rejection reasons, and how to address reviewer concerns.

Last Updated: 2024

Overview

Reviewers at different venues prioritize different aspects. Understanding these priorities helps you:

1. Frame your contribution appropriately
2. Anticipate likely criticisms
3. Prepare effective rebuttals
4. Decide where to submit

High-Impact Journals (Nature, Science, Cell)

What Reviewers Look For

Review Process

1. Editorial triage: Most papers rejected without review (Nature: ~92%)
2. Expert review: 2-4 reviewers if sent out
3. Cross-discipline reviewer: Often includes non-specialist
4. Quick turnaround: First decision typically 2-4 weeks

What Gets a Paper Rejected

At Editorial Stage:

• Findings not significant enough for broad audience
• Incremental advance over prior work
• Too specialized for the journal
• Topic doesn't fit current editorial interests

At Review Stage:

• Claims not supported by data
• Missing critical controls
• Alternative interpretations not addressed
• Statistical concerns
• Prior work not adequately acknowledged
• Writing inaccessible to non-specialists

How to Address Nature/Science Reviewers

In the paper:

• Lead with significance in the first paragraph
• Explain why findings matter broadly
• Include controls for all major claims
• Use clear, accessible language
• Include conceptual figures

In rebuttal:

• Address every point (even minor ones)
• Provide new data when requested
• Acknowledge valid criticisms gracefully
• Explain significance if questioned

Sample Reviewer Concerns and Responses

Reviewer: "The significance of this work is unclear to a general audience."

Response: "We have revised the introduction to clarify the broader significance. As now stated in paragraph 1, our findings have implications for [X] because [Y]. We have also added a discussion of how these results inform understanding of [Z] (p. 8, lines 15-28)."

Medical Journals (NEJM, Lancet, JAMA)

What Reviewers Look For

Review Process

1. Statistical review: Dedicated statistical reviewer common
2. Clinical expertise: Subspecialty experts
3. Methodological review: Focus on study design
4. Multiple rounds: Revisions often requested

What Gets a Paper Rejected

Major Issues:

• Underpowered study
• Inappropriate control/comparator
• Confounding not addressed
• Selective outcome reporting
• Missing safety data
• Claims exceed evidence

Moderate Issues:

• Unclear generalizability
• Missing subgroup analyses
• Incomplete CONSORT/STROBE reporting
• Statistical methods not described adequately

Sample Reviewer Concerns and Responses

Reviewer: "The study appears underpowered for the primary outcome. With 200 participants and an event rate of 5%, there is insufficient power to detect a clinically meaningful difference."

Response: "We appreciate this concern. Our power calculation (Methods, p. 5) was based on a 5% event rate in the control arm and a 50% relative reduction (to 2.5%). While the observed event rate (4.8%) was close to projected, we acknowledge the confidence interval is wide (HR 0.65, 95% CI 0.38-1.12). We have added this as a limitation (Discussion, p. 12). Importantly, the direction and magnitude of effect are consistent with the larger XYZ trial (n=5000), suggesting our findings merit confirmation in a larger study."

Cell Press Journals

What Reviewers Look For

Review Process

1. Extended review: 3+ reviewers typical
2. Revision cycles: Multiple rounds common
3. Comprehensive revision: Major new experiments often requested
4. Detailed assessment: Figure-by-figure evaluation

What Reviewers Expect

• Multiple complementary approaches: Same finding shown different ways
• In vivo validation: For cell biology claims
• Rescue experiments: For knockdown/knockout studies
• Quantification: Not just representative images
• Complete figure panels: All conditions, all controls

Sample Reviewer Concerns and Responses

Reviewer: "The authors show that protein X is required for process Y using siRNA knockdown. However, a single RNAi reagent is used, and off-target effects cannot be excluded. Additional evidence is needed."

Response: "We agree that additional validation is important. In the revised manuscript, we now show: (1) two independent siRNAs against protein X produce identical phenotypes (new Fig. S3A-B); (2) CRISPR-Cas9 knockout cells recapitulate the phenotype (new Fig. 2D-E); and (3) expression of siRNA-resistant protein X rescues the phenotype (new Fig. 2F-G). These complementary approaches strongly support the conclusion that protein X is required for process Y."

ML Conferences (NeurIPS, ICML, ICLR)

What Reviewers Look For

Review Process

1. Area Chair assignment: Grouped by topic
2. 3-4 reviewers: With expertise in the area
3. Author rebuttal: Opportunity to respond
4. Reviewer discussion: After rebuttal
5. AC recommendation: Meta-review

Scoring Dimensions

Typical NeurIPS/ICML scoring:

What Gets a Paper Rejected

Critical Issues:

• Weak baselines or unfair comparisons
• Missing ablation studies
• Results not significantly better than SOTA
• Technical errors in method or analysis
• Overclaiming without evidence

Moderate Issues:

• Limited novelty over prior work
• Narrow evaluation (few datasets/tasks)
• Missing reproducibility details
• Poor presentation
• Limited analysis or insights

Red Flags for ML Reviewers

❌ "We compare against methods from 2018" (outdated baselines) ❌ "Our method achieves 0.5% improvement" (marginal gain) ❌ "We evaluate on one dataset" (limited generalization) ❌ "Implementation details are in the supplementary" (core info missing) ❌ "We leave ablations for future work" (incomplete evaluation)

Sample Reviewer Concerns and Responses

Reviewer: "The proposed method is only compared against Transformer and Performer. Recent works like FlashAttention and Longformer should be included."

Response: "Thank you for this suggestion. We have added comparisons to FlashAttention (Dao et al., 2022), Longformer (Beltagy et al., 2020), and BigBird (Zaheer et al., 2020). As shown in new Table 2, our method outperforms all baselines: FlashAttention (3.2% worse), Longformer (5.1% worse), and BigBird (4.8% worse). We also include a new analysis (Section 4.3) explaining why our approach is particularly effective for sequences > 16K tokens."

HCI Conferences (CHI, CSCW)

What Reviewers Look For

Contribution Types

CHI explicitly categorizes contributions:

What Gets a Paper Rejected

Critical Issues:

• Mismatch between claims and evaluation
• Insufficient participants for conclusions
• Missing ethical considerations (no IRB)
• Technology-focused without user insight
• Limited contribution to HCI community

Moderate Issues:

• Weak design rationale
• Limited generalizability
• Missing related work in HCI
• Unclear implications for practitioners

Sample Reviewer Concerns and Responses

Reviewer: "The evaluation consists of a short-term lab study with 12 participants. It's unclear how this system would perform in real-world use over time."

Response: "We acknowledge this limitation, which we now discuss explicitly (Section 7.2). We have added a 2-week deployment study with 8 participants from our original cohort (new Section 6.3). This longitudinal data shows sustained engagement (mean usage: 4.2 times/day) and reveals additional insights about how use patterns evolve over time. However, we agree that larger and longer deployments would strengthen ecological validity."

NLP Conferences (ACL, EMNLP)

What Reviewers Look For

ACL Rolling Review (ARR)

Since 2022, ACL venues use a shared review system:

• Reviews transfer between venues
• Action editors manage papers
• Commitment to specific venue after review

Responsible NLP Checklist

Reviewers check for:

• Limitations section (required)
• Risks and ethical considerations
• Compute/carbon footprint
• Bias analysis (when applicable)
• Data documentation

Sample Reviewer Concerns and Responses

Reviewer: "The paper lacks analysis of failure cases. When and why does the proposed method fail?"

Response: "We have added Section 5.4 on error analysis. We manually examined 100 errors and categorized them into three types: (1) complex coreference chains (42%), (2) implicit references (31%), and (3) domain-specific knowledge requirements (27%). Figure 4 shows representative examples of each. This analysis reveals that our method particularly struggles with implicit references, which we discuss as a direction for future work."

Data Mining (KDD, WWW)

What Reviewers Look For

What Impresses KDD Reviewers

• Large-scale experiments (millions of samples)
• Industry deployment or A/B tests
• Efficiency comparisons (runtime, memory)
• Real datasets alongside benchmarks
• Complexity analysis (time and space)

Sample Reviewer Concerns and Responses

Reviewer: "The experiments are limited to small datasets (< 100K samples). How does the method scale to industry-scale data?"

Response: "We have added experiments on two large-scale datasets: (1) ogbn-papers100M (111M nodes, 1.6B edges) and (2) a proprietary e-commerce graph (500M nodes, 4B edges) provided by [company]. Table 4 (new) shows our method scales near-linearly with data size, completing in 42 minutes on ogbn-papers where baselines run out of memory. Section 5.5 (new) provides detailed scalability analysis."

General Rebuttal Strategies

Do's

✅ Address every point: Even minor issues ✅ Provide evidence: New experiments, data, or citations ✅ Be specific: Reference exact sections, lines, figures ✅ Acknowledge valid criticisms: Show you understand the concern ✅ Be concise: Reviewers read many rebuttals ✅ Stay professional: Even for unfair reviews ✅ Prioritize critical issues: Address major concerns first

Don'ts

❌ Be defensive: Accept valid criticisms ❌ Argue without evidence: Back up claims ❌ Ignore points: Even ones you disagree with ❌ Be vague: Be specific about changes ❌ Attack reviewers: Maintain professionalism ❌ Promise future work: Do the work now if possible

Rebuttal Template

We thank the reviewers for their constructive feedback. We address 
the main concerns below:

**R1/R2 Concern: [Shared concern from multiple reviewers]**

[Your response with specific actions taken and references to where 
changes are made in the revised manuscript]

**R1-1: [Specific point]**

[Response with evidence]

**R2-3: [Specific point]**

[Response with evidence]

We have also made the following additional improvements:
• [Improvement 1]
• [Improvement 2]

Pre-Submission Self-Review

Before submitting, review your paper as a reviewer would:

All Venues

• Are claims supported by evidence?
• Are baselines appropriate and recent?
• Is the contribution clearly stated?
• Are limitations acknowledged?
• Is reproducibility information complete?

High-Impact Journals

• Is significance clear to a non-specialist?
• Are figures accessible and clear?
• Are controls adequate for claims?

Medical Journals

• Is CONSORT/STROBE compliance complete?
• Are absolute numbers reported?
• Is clinical relevance clear?

ML Conferences

• Are ablations comprehensive?
• Are comparisons fair?
• Is reproducibility information complete?

HCI Conferences

• Is the user-centered perspective clear?
• Is the evaluation appropriate for claims?
• Are design implications actionable?

See Also

• venue_writing_styles.md - Writing style by venue
• nature_science_style.md - Nature/Science detailed guide
• ml_conference_style.md - ML conference detailed guide
• medical_journal_styles.md - Medical journal detailed guide