Re2 Peer Review Eval

re2-peer-review-eval

Re$^2$: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions — Zhang et al. (2025) (arXiv:2505.07920, 2025)

What this evaluates

Evaluates LLM capabilities across the full academic peer review lifecycle, including predicting paper acceptance and scores, generating structured peer reviews, and simulating multi-turn author-reviewer rebuttal conversations.

Datasets

• Re$^2$ — total 19926; splits: train (-1), test (1000), test_rebuttal (500)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified acceptance/rejection decisions out of total test instances.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for the binary acceptance prediction task.
• MAE — range: [0, inf)
  • Mean Absolute Error between predicted and actual review scores.
• MSE — range: [0, inf)
  • Mean Squared Error between predicted and actual review scores.
• BLEU — range: [0, 100]
  • N-gram based lexical overlap metric between generated reviews/rebuttals and reference texts.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap metric between generated and reference texts.
• BERTScore — range: [0, 1]
  • Semantic similarity computed using cosine similarity of contextual embeddings from DeBERTa-large-MNLI.
• EmbedCos — range: [0, 1]
  • Cosine similarity between sentence embeddings generated by sentence-transformers/all-mpnet-base-v2.
• LLM-judge scores — range: [0, 10]
  • Scores assigned by LLaMA-3.1-8B-Instruct judging accuracy, constructiveness, completeness, clarity, and quality of rebuttal responses.

Input / output format

Input: Paper content/abstract (for prediction and review generation); paper content plus reviewer comment (for rebuttal generation).

Output: Binary acceptance decision or numerical score; generated review text; generated rebuttal response text.

Scoring recipe

def compute_metrics(predictions, gold):
    # Acceptance Prediction
    acc = (predictions == gold).mean()
    prec, rec, f1 = precision_recall_fscore(gold, predictions)
    
    # Score Prediction
    mae = mean_absolute_error(gold, predictions)
    mse = mean_squared_error(gold, predictions)
    
    # Generation Metrics
    bleu = compute_bleu(gold, predictions)
    rouge_l = compute_rouge_l(gold, predictions)
    bertscore = compute_bertscore(gold, predictions)
    embedcos = cosine_similarity(embed(gold), embed(predictions))
    
    # LLM Judge
    judge_scores = llm_judge(gold, predictions, prompt=JUDGE_PROMPT)
    return {'accuracy': acc, 'F1': f1, 'MAE': mae, 'MSE': mse, 'BLEU': bleu, 'ROUGE-L': rouge_l, 'BERTScore': bertscore, 'EmbedCos': embedcos, 'Judge': judge_scores}

Common pitfalls

• Zero-shot LLMs exhibit a strong bias toward accepting all papers, yielding artificially high recall but meaningless precision and F1 scores.
• LLM-as-judge evaluations rely on a single model (LLaMA-3.1-8B-Instruct) and specific prompts, which may not align with human reviewer standards or introduce systematic bias.
• EmbedCos and BERTScore measure semantic similarity but do not verify factual correctness or adherence to specific review guidelines.

Evidence (verbatim from paper)

The acceptance prediction task aims to predict whether a paper will be accepted or rejected based on its content. It is a two-class classification task (acceptance or rejection), so we use accuracy, precision, recall, and F1 score as the evaluation metrics of the acceptance prediction task. Further, score prediction focuses not just on acceptance outcomes, but on predicting the detailed review scores (e.g., overall rating) that a paper would receive. Its performance is typically measured using Mean Absolute Error (MAE) and Mean Squared Error (MSE).

Citation

@misc{zhang2025re2,
  title={Re$^2$: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2505.07920}
}

• arXiv: 2505.07920