Review Quality Eval

review-quality-eval

Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time — Kuznetsov et al. (2026) (arXiv:2601.15172, 2026)

What this evaluates

Evaluates the quality of academic peer review reports across different conferences and years using a multi-dimensional framework. It measures how substantive, actionable, and well-grounded reviews are, and tracks whether these qualities decline over time.

Datasets

• Peer Review Campaigns (ICLR, NeurIPS, ACL) — total ?; splits: full (-1); repo https://github.com/UKPLab/arxiv2026-review-quality-estimation

Metrics

• Q (primary) — range: [0, 1]
  • Non-weighted average of all normalized measurement scores (LEN, ITX, EXL, GND, ACT, etc.) per review. Each base metric is first normalized to [0,1].
• Spearman's ρ — range: [-1, 1]
  • Rank-based correlation coefficient used to measure monotonic relationships between metric pairs and between lightweight vs LLM-based aggregate scores.

Input / output format

Input: Raw peer review text (30 to 28k characters) associated with a submitted paper.

Output: Normalized scores for each quality dimension and an aggregate quality score Q per review.

Scoring recipe

# Normalize each base metric m to [0,1]
m_norm = (m - m_min) / (m_max - m_min)
# Compute aggregate score Q
Q = mean([m_norm for m in base_metrics])
# Correlation
rho = spearmanr(scores_A, scores_B)
# Significance test
p_val = bootstrap_test(Q_year1, Q_year2, n_iter=10000, ci=0.99, correction='bonferroni')

Common pitfalls

• Count-based metrics (LEN, core items) have long tails and positive skew, requiring careful normalization before aggregation.
• Low Q scores may reflect non-genuine reviews (e.g., placeholder text, refusal to review, desk-reject flags) rather than poor quality.
• LLM-based metrics are computationally expensive and raise confidentiality concerns, so lightweight proxies are used for large-scale analysis.

Evidence (verbatim from paper)

Based on the effective ranges of the metrics, we normalize each metric to $[0,1]$ for ease of reporting and aggregation. Given that none of the metric pairs show extremely high correlations ($ ho$ > 0.9), we define an aggregate review quality score $Q$ as a non-weighted average of all our measurements (after normalization).

Citation

@misc{kuznetsov2026peerreview,
  title={Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time},
  author={Kuznetsov et al. (2026)},
  year={2026},
  note={arXiv:2601.15172}
}

• arXiv: 2601.15172