Treereview Peer Review Eval

treereview-peer-review-eval

TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review — Yuan Chang et al. (arXiv:2506.07642, 2025)

What this evaluates

This benchmark evaluates an LLM's ability to perform deep, structured scientific peer review by generating comprehensive reviews and actionable feedback comments. It probes the model's capacity for hierarchical question decomposition, context-aware analysis of long documents, and alignment with human reviewer judgments across multiple quality dimensions.

Datasets

• TreeReview Benchmark (ICLR-2024, NeurIPS-2023, Nature Communications) — total 120; splits: test (120); repo https://github.com/YuanChang98/tree-review

Metrics

• Overall Quality (LLM-as-Judge) (primary) — range: [0, 10]
  • Gemini-2.5-Pro rates generated reviews on a 0-10 scale across eight dimensions (Comprehensiveness, Technical Depth, Clarity, Constructiveness, Specificity, Evidence Support, Consistency, Overall Quality). Scores are averaged over three independent runs with temperature 0.1.
• LLM-based alignment Precision — range: percent
  • Many-to-many matching between generated and reference comments using Gemini-2.5-Pro. Precision is the proportion of generated comments judged as highly related to the reference set.
• MAE (Rating Alignment) — range: other
  • Mean Absolute Error between system-assigned numerical ratings (Soundness, Presentation, Contribution, Overall) and the average ground-truth human ratings.
• ITF-IDF — range: other
  • Measures the diversity and uniqueness of content in generated feedback comments. Higher scores indicate more specific and less generic feedback.

Input / output format

Input: Scientific paper (text/PDF). For the proposed method, papers are chunked into 1024-token segments, with the top-3 most relevant chunks selected as context per leaf question. Baselines receive full paper text or structured guidelines.

Output: Task 1: Comprehensive review text (summary, strengths, weaknesses, questions) plus numerical ratings (Soundness, Presentation, Contribution, Overall). Task 2: List of specific, critical feedback comments targeting substantive weaknesses.

Scoring recipe

def evaluate(predictions, gold):
    # LLM-as-Judge scoring
    scores = []
    for _ in range(3):
        judge = llm_judge(predictions, temp=0.1)
        scores.append(judge['Overall Quality'])
    overall_quality = sum(scores) / len(scores)
    
    # LLM-based alignment for comments
    pred_comments = extract_comments(predictions)
    ref_comments = merge_human_reviews(gold)
    matches = llm_match_many_to_many(pred_comments, ref_comments)
    precision = len(matches) / len(pred_comments) if pred_comments else 0
    
    return {'Overall Quality': overall_quality, 'Alignment Precision': precision}

Common pitfalls

• Using ROUGE or BERTScore for evaluation, which the authors explicitly note fail to capture nuanced review qualities.
• Averaging LLM judge scores over only one run instead of three independent runs with temperature 0.1.
• Treating individual human reviews as separate ground truths instead of merging them into a single integrated reference set for comment alignment.

Evidence (verbatim from paper)

Specifically, we implement a score-based evaluation procedure using Gemini-2.5-Pro (version gemini-2.5-pro-exp-0325) to rate system-generated reviews on a 0-10 scale across eight dimensions: Comprehensiveness, Technical Depth, Clarity, Constructiveness, Specificity, Evidence Support, Consistency, and the Overall Quality. This approach enables more meaningful and fine-grained quality assessment of reviews. To ensure reliable evaluation, we conduct three independent scoring runs with temperature 0.1 and average the results as final scores.

Citation

@misc{chang2025treereview,
  title={TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review},
  author={Yuan Chang et al.},
  year={2025},
  note={arXiv:2506.07642}
}

• arXiv: 2506.07642