qhjqhj00 peer-review-analysis-eval
Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI — Wu et al. (2026) (arXiv:2604.19578, 2026)
What this evaluates
This protocol evaluates the linguistic and content-level properties of academic peer review reports to assess how LLM assistance influences review quality, complexity, and aspect coverage over time.
Datasets
- ICLR & NeurIPS Peer Reviews — total ?; splits: ICLR 2017-2025 (-1), NeurIPS 2016-2024 (-1); repo https://github.com/njust-winchy/LLM_impact
Metrics
text_length— range: count- Average number of sentences and words per review report.
linguistic_complexity— range: proportion- Measures lexical richness and syntactic indicators such as nominal subjects, auxiliary verbs, direct objects, and adverbial clauses per clause.
aspect_mentions(primary) — range: mixed- Average length of aspect-related sentences, average number of aspect mentions (e.g., Clarity, Soundness, Originality), and sentiment polarity distribution for each aspect.
llm_detection_rate— range: [0, 1]- Probability of a review being LLM-assisted estimated via maximum likelihood estimation.
Input / output format
Input: Peer review reports from ICLR and NeurIPS conferences, including full text content and reviewer self-reported confidence scores (1-5).
Output: Computed statistics per year/conference: average text length, lexical/syntactic complexity metrics, aspect mention counts and sentiments, and LLM-assistance probability scores.
Scoring recipe
def compute_metrics(reviews):
stats = {}
for review in reviews:
stats['text_len'] = count_sentences_and_words(review.text)
stats['aspects'] = extract_aspect_mentions(review.text)
stats['sentiment'] = compute_sentiment_polarity(review.text)
stats['confidence'] = review.confidence_score
stats['llm_prob'] = max_likelihood_estimate_llm_assistance(review.text)
return aggregate_by_year_and_confidence(stats)
Common pitfalls
- Temporal boundaries for 'pre-LLM' vs 'post-LLM' are defined by conference dates (ICLR 2023, NeurIPS 2022) rather than model release dates.
- Aspect mentions are identified via NLP pipelines rather than manual annotation, which may affect sentiment polarity accuracy.
- Confidence scores are self-reported by reviewers and may not correlate linearly with review quality or actual LLM usage.
Evidence (verbatim from paper)
Specifically, we analyze: The average length of aspect-related contents to measure elaboration depth (Figure 5). The average number of aspect mentions (e.g., Clarity, Soundness, Originality) to assess topic coverage (Figure 6). The average sentiment polarity associated with each aspect to explore tonal tendencies (Figure 7).
Citation
@misc{wu2026impact,
title={Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI},
author={Wu et al. (2026)},
year={2026},
note={arXiv:2604.19578}
}
- arXiv: 2604.19578