qhjqhj00 iclr2025-review-feedback-eval
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 — Nitya Thakkar et al. (2025) (arXiv:2504.09737, 2025)
What this evaluates
Evaluates the causal impact of LLM-generated feedback on peer review quality by measuring reviewer engagement, revision rates, feedback incorporation, and downstream rebuttal dynamics in a large-scale randomized controlled trial.
Datasets
- ICLR 2025 Reviews — total 44831; splits: control (22364), feedback (22467); repo https://github.com/zou-group/review_feedback_agent
Metrics
update_rate(primary) — range: percent- Percentage of reviews that are edited after receiving feedback. An update is defined as an edit distance greater than 5 between the initial and modified review. For the control group, updates must occur at least one hour after the initial post to account for feedback latency.
incorporation_rate— range: percent- Proportion of LLM-generated feedback items that are clearly integrated into the reviewer's modified review, measured via an LLM-based pipeline validated against human annotations.
review_length_change— range: words- Average difference in word count between initial and modified reviews, calculated across the summary, strengths, weaknesses, and questions sections.
acceptance_rate— range: percent- Percentage of papers accepted in the control group versus the feedback group, where groups are defined by whether any review for the paper was selected to receive feedback.
Input / output format
Input: Initial peer review text (summary, strengths, weaknesses, questions) and paper metadata.
Output: Modified peer review text, updated numerical scores, and subsequent rebuttal/reply text.
Scoring recipe
def calculate_update_rate(initial_reviews, modified_reviews):
updated_count = 0
for init, mod in zip(initial_reviews, modified_reviews):
if edit_distance(init, mod) > 5:
updated_count += 1
return updated_count / len(initial_reviews) * 100
def calculate_incorporation_rate(feedback_items, modified_reviews):
incorporated = llm_pipeline.check_integration(feedback_items, modified_reviews)
return incorporated / len(feedback_items) * 100
Common pitfalls
- Using a strict edit distance threshold (<5) to filter out minor typo fixes or score-only changes, which significantly affects update rate calculations.
- Failing to account for the intent-to-treat design, where 7.9% of the feedback group did not actually receive feedback, diluting the measured effect size.
- Comparing control group updates without enforcing a >1 hour delay, which would conflate natural late edits with feedback-induced updates.
Evidence (verbatim from paper)
With an update rate difference of roughly 17 percentage points (Figure 3B), we can see that reviews that received feedback were much more likely to be updated than those that did not.
Citation
@misc{thakkar2025canllmfeedbackenhance,
title={Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025},
author={Nitya Thakkar et al. (2025)},
year={2025},
note={arXiv:2504.09737}
}
- arXiv: 2504.09737