qhjqhj00 iclr-ai-review-impact-eval
The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates — Russo Latona et al. (2024) (arXiv:2405.02150, 2024)
What this evaluates
Probes the causal impact of LLM-assisted peer reviews on paper scoring and acceptance outcomes at a major machine learning conference. It measures whether AI-assisted reviews systematically inflate scores and increase acceptance probabilities, particularly for borderline submissions.
Datasets
- ICLR Conference Reviews (2018-2024) — total ?; splits: 2018-2023 (-1), 2024 (28028); repo https://github.com/epfl-dlab/AIRewiewLottery
Metrics
mean_score_difference— range: other- Average of (r_ai - r_h) across all valid review pairs per paper, where r_ai is AI-assisted score and r_h is human score.
acceptance_rate_difference(primary) — range: percent- Coefficient β from linear regression y_k = α + β·L_k + γ·X_k on a matched sample, representing percentage point change in acceptance probability.
relative_probability_difference— range: percent- From ordinal regression: probability that an AI-assisted review scores higher than a human review given they differ, minus 50%.
Input / output format
Input: Per submission: review texts, review scores, abstract text, and final acceptance decision. Per review: text for LLM detection.
Output: Binary AI-assisted/human label per review; aggregated statistical estimates (mean score difference, regression coefficients for acceptance).
Scoring recipe
# 1. Classify reviews using GPTZero (prob < 0.5 => AI-assisted)
ai_reviews = [r for r in reviews if gptzero_prob(r) < 0.5]
human_reviews = [r for r in reviews if gptzero_prob(r) >= 0.5]
# 2. Compute mean score difference (Study 2)
diffs = []
for paper in papers_with_at_least_3_reviews:
for ai_r in paper.ai_reviews:
for h_r in paper.human_reviews:
if h_r != reference_review:
diffs.append(ai_r.score - h_r.score)
mean_diff = np.mean(diffs)
# 3. Compute acceptance_rate_difference (Study 3)
matched_pairs = match_submissions_by_review_count_and_scores()
beta = fit_linear_regression(matched_pairs, target='acceptance', features=['has_ai_review', 'controls']).coef['has_ai_review']
Common pitfalls
- GPTZero's false-negative rate is uncorrected, making prevalence estimates a lower bound.
- Review scores are ordinal, yet treated as continuous for mean difference calculations.
- Matching relies on semantic similarity of abstracts/reviews, which may not fully control for paper quality or topic-specific reviewer biases.
Evidence (verbatim from paper)
We find that, on average, AI-assisted reviews were 0.14 points (95% CI [0.08, 0.19]) higher than human reviews. ... The fitted coefficients reveal that submissions that received AI-assisted reviews had 13.8% higher odds of being accepted compared to those that did not (p = 0.024), or alternatively, had 3.1 percentage points higher chances of being accepted (p = 0.024).
Citation
@misc{russo2024aireviewlottery,
title={The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates},
author={Russo Latona et al. (2024)},
year={2024},
note={arXiv:2405.02150}
}
- arXiv: 2405.02150