Reviewmt Eval

reviewmt-eval

Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions — Tan et al. (2024) (arXiv:2406.05688, 2024)

What this evaluates

Evaluates LLMs on simulating the academic peer review process across multi-turn dialogues. It probes the model's ability to generate relevant paper summaries, write comprehensive reviews, and make accurate acceptance or rejection decisions based on long-context interactions.

Datasets

• ReviewMT — total ?; splits: train (-1), test (100); repo https://github.com/chengtan9907/ReviewMT

Metrics

• Paper hit rate — range: percent
  • Percentage of test instances where the generated paper summary exactly matches or sufficiently overlaps with the ground truth summary.
• Review hit rate — range: percent
  • Percentage of test instances where the generated review text exactly matches or sufficiently overlaps with the ground truth review.
• Decision hit rate — range: percent
  • Percentage of test instances where the model's predicted acceptance/rejection decision exactly matches the ground truth decision.
• MAE — range: other
  • Mean Absolute Error between the model's predicted numerical score and the ground truth score. Only reported for the multi-turn ICLR subset.
• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for the binary decision classification task (accept vs. reject).

Input / output format

Input: Paper text and multi-turn dialogue history (author responses, reviewer comments, decision maker feedback).

Output: Generated paper summary, review text, and acceptance/rejection decision.

Scoring recipe

def compute_metrics(preds, golds):
    n = len(preds)
    paper_hits = sum(1 for p, g in zip(preds['paper'], golds['paper']) if exact_match(p, g))
    review_hits = sum(1 for p, g in zip(preds['review'], golds['review']) if exact_match(p, g))
    decision_hits = sum(1 for p, g in zip(preds['decision'], golds['decision']) if p == g)
    maes = [abs(p - g) for p, g in zip(preds['score'], golds['score'])]
    f1 = f1_score(golds['decision'], preds['decision'], average='macro')
    return {
        'paper_hit_rate': paper_hits / n,
        'review_hit_rate': review_hits / n,
        'decision_hit_rate': decision_hits / n,
        'mae': sum(maes) / n,
        'f1_score': f1
    }

Common pitfalls

• Zero-shot models show very low review and decision hit rates, requiring supervised fine-tuning for competitive performance.
• MAE and review hit rates are only evaluated on the multi-turn ICLR subset, not the one-turn NC subset.
• Some models (e.g., Yuan) have strict safety constraints that prevent them from outputting decisions, artificially lowering their decision hit rate.

Evidence (verbatim from paper)

As shown in Table[2], most LLMs demonstrate high paper hit rates, indicating their ability to generate relevant content related to the papers. However, zero-shot performance reveals lower review hit rates and decision hit rates, suggesting that LLMs struggle to provide scores and decisions. Supervised fine-tuning significantly improves performance across all metrics.

Citation

@misc{tan2024reviewmt,
  title={Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions},
  author={Tan et al. (2024)},
  year={2024},
  note={arXiv:2406.05688}
}

• arXiv: 2406.05688