Reviewer Too Eval

reviewer-too-eval

ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review — Sahu et al. (2025) (arXiv:2510.08867, 2025)

What this evaluates

Evaluates the ability of LLM-based reviewer agents to predict conference acceptance decisions and generate high-quality peer reviews. It probes classification accuracy against human decisions and assesses review quality through LLM-judged pairwise comparisons across multiple dimensions.

Datasets

• ICLR-2k dataset — total 1963; splits: test (1963)

Metrics

• macro-F1 (5-way) (primary) — range: [0, 1]
  • Macro-averaged F1 score across five decision categories (Oral, Spotlight, Poster, Reject, Desk Reject), computed by averaging per-class F1 scores.
• Accept/Reject Accuracy — range: [0, 1]
  • Overall accuracy on the binary accept/reject classification task, calculated as the proportion of correctly predicted decisions.
• ELO rating — range: other
  • Pairwise win/loss/draw outcomes from LLM judges across five quality axes (Depth, Actionability, Summary, Clarity, Helpfulness), aggregated into an ELO score using standard logistic updates.

Input / output format

Input: Manuscript text, optionally supplemented with conference guidelines, author rebuttals, and retrieved literature summaries depending on the baseline configuration.

Output: A predicted conference decision category (Oral, Spotlight, Poster, Reject, or Desk Reject) and/or a structured review text.

Scoring recipe

def compute_macro_f1(predictions, gold, classes):
    prec, rec, f1, _ = precision_recall_fscore_support(gold, predictions, average='macro', labels=classes)
    return f1

def compute_accuracy(predictions, gold):
    correct = sum(p == g for p, g in zip(predictions, gold))
    return correct / len(gold)

Common pitfalls

• Withdrawn papers are merged into the Reject category, artificially inflating rejection rates compared to standard conference splits.
• Review quality relies on LLM judges rather than human experts, which may introduce bias in axes like Depth or Actionability.
• The dataset is stratified by original average reviewer scores, limiting generalizability to unreviewed or out-of-distribution submissions.

Evidence (verbatim from paper)

We assess alignment with real conference decisions by measuring both the 5-way classification performance (Oral, Spotlight, Poster, Reject, Desk Reject) and the binary Accept/Reject task; we report macro-averaged Precision, Recall, and F1, with macro averaging across classes $c$. We also report overall Accuracy, and False Positive Rate (for binary task).

Citation

@misc{sahu2025reviewertoo,
  title={ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review},
  author={Sahu et al. (2025)},
  year={2025},
  note={arXiv:2510.08867}
}

• arXiv: 2510.08867