LLM Peer Review Eval

llm-peer-review-eval

From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review — Zhang et al. (2025) (arXiv:2506.11343, 2025)

What this evaluates

Evaluates whether an LLM-based pairwise comparison framework can effectively identify high-impact academic papers compared to human peer review and traditional rating-based LLM methods. It probes the system's predictive accuracy for future scholarly influence, decision consistency with human committees, and susceptibility to biases in topic novelty and institutional representation.

Datasets

• OpenReview Conference Papers (ICLR, NeurIPS, CoRL, EMNLP) — total ?; splits: test (-1)

Metrics

• average_citation_count (primary) — range: other
  • Mean number of citations received by papers accepted by the system. Used as the primary proxy for academic impact and system performance scaling.
• decision_consistency — range: percent
  • Percentage of papers where the GPT ranking system's decision (Oral, Spotlight, Poster, or Reject) matches the human peer review decision.
• topic_novelty_distance — range: other
  • Mean Euclidean distance between each paper's abstract embedding and its nearest neighbor within the same conference corpus. Smaller distances indicate lower novelty.
• institutional_inequality_gini — range: [0, 1]
  • Gini coefficient calculated over the distribution of first-author affiliations among top-tier accepted papers. Higher values indicate greater concentration at elite institutions.

Input / output format

Input: Extracted paper content including title, abstract, figure/table captions, and main text, obtained via ScienceBeam from conference PDFs.

Output: Acceptance decision categorized into one of four tiers: Oral, Spotlight, Poster, or Reject, derived from aggregated pairwise comparison scores.

Scoring recipe

def compute_average_citation_count(predictions, gold_citations):
    accepted_citations = [gold_citations[pid] for pid in predictions if pid in gold_citations]
    if not accepted_citations:
        return 0.0
    return sum(accepted_citations) / len(accepted_citations)

def compute_decision_consistency(gpt_decisions, human_decisions):
    matches = sum(1 for g, h in zip(gpt_decisions, human_decisions) if g == h)
    return (matches / len(gpt_decisions)) * 100

Common pitfalls

• Confusing the proposed pairwise ranking system with the GPT rating system baseline, which uses independent reviewers and a meta-reviewer instead of direct comparisons.
• Treating citation count as a direct quality metric without accounting for field-specific citation norms, conference tier differences, or citation lag across years.
• Assuming the embedding-based novelty distance perfectly captures conceptual innovation, as it only measures semantic similarity to existing abstracts.

Evidence (verbatim from paper)

We now empirically validate whether our proposed pairwise-based ranking framework better identifies high-impact papers compared to conventional rating-based approaches. Figure 2 shows the overall academic impact (measured by their average citation counts) of the accepted papers as the number of agents increases.

Citation

@misc{zhang2025pairwisepeerreview,
  title={From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2506.11343}
}

• arXiv: 2506.11343