Review Quality Metrics

review-quality-metrics

ML Researchers Support Openness in Peer Review But Are Concerned About Resubmission Bias — Rao et al. (2025) (arXiv:2511.23439, 2025)

What this evaluates

Evaluates the quality of academic peer reviews by measuring how well criticisms are supported by evidence (substantiation), how factually accurate the review's claims are (correctness), and how thoroughly the review covers the paper's contributions (completeness).

Datasets

• ICLR 2023 & NeurIPS 2022 accepted papers & reviews — total ?; splits: test (-1); repo https://github.com/justinpayan/OpenReviewAnalysis

Metrics

• substantiation — range: [0, 1]
  • Fraction of negative evaluative segments in a review that are well-substantiated with evidence or examples from the paper.
• correctness (primary) — range: [0, 1]
  • Fraction of objective segments in a review that are factually correct, verified against the paper text and author responses.
• completeness — range: [0, 1]
  • Fraction of all claims (contributions/outputs) made in the paper that are mentioned in some form in the review.

Input / output format

Input: Per instance: the full text of a research paper, the full text of its peer review, and (for correctness evaluation) the author responses to that review.

Output: Per instance: aggregated scores for substantiation, correctness, and completeness, each ranging from 0 to 1, representing the fraction of substantiated negative segments, correct objective segments, and covered paper claims, respectively.

Scoring recipe

# Substantiation
neg_segs = extract_negative_evaluative_segments(review)
sub_score = sum(1 for s in neg_segs if is_substantiated(s, paper)) / len(neg_segs)

# Correctness
obj_segs = extract_objective_segments(review)
corr_score = sum(1 for s in obj_segs if is_correct(s, paper, author_responses)) / len(obj_segs)

# Completeness
claims = extract_claims_from_paper(paper)
comp_score = sum(1 for c in claims if is_covered(c, review)) / len(claims)

Common pitfalls

• Excluding the review summary section is critical for completeness, as it often just paraphrases the abstract and artificially inflates scores.
• Author responses must be included when evaluating correctness, as authors may have added experiments or clarifications after the review was written, making seemingly incorrect statements actually correct.
• Only negative evaluative segments are used for substantiation, as positive praise does not require detailed evidence.

Evidence (verbatim from paper)

We define three key metrics that serve as a measure of the quality of the review. The first is substantiation, which is a measure of how well the review is supported with evidence and examples from the paper. Substantiation is important as it tells the authors what needs to be changed in the paper for acceptance (either in the current venue or in the future). When a reviewer substantiates the review well, it shows that they are thinking through their decision making recommendations. The second is correctness, which measures the level of factuality in the review. Correctness is naturally a key metric, since an incorrect analysis of the paper could directly result in the wrongful acceptance or rejection of a paper. The third metric is completeness, which is a measure of how many of the paper's contributions have been covered in the review. It is therefore an indication of whether the reviewer has missed something.

Citation

@misc{rao2025openreviewanalysis,
  title={ML Researchers Support Openness in Peer Review But Are Concerned About Resubmission Bias},
  author={Rao et al. (2025)},
  year={2025},
  note={arXiv:2511.23439}
}

• arXiv: 2511.23439