Echoreview Bench Eval

echoreview-bench-eval

EchoReview: Learning Peer Review from the Echoes of Scientific Citations — Zhang et al. (2026) (arXiv:2602.00733, 2026)

What this evaluates

Evaluates the quality, comprehensiveness, and evidence support of AI-generated academic peer reviews across multiple dimensions. It also measures the alignment between AI-identified research limitations and human reviewer findings, as well as the impact of citation time spans on review coherence and technical focus.

Datasets

• EchoReview-Bench — total ?; splits: test_a (-1), test_b (233)

Metrics

• Overall Quality score (primary) — range: [0, 10]
  • Mean of scores (0-10) across Comprehensiveness, Specificity, Evidence Support, and Consistency. Each review is scored three times by an LLM judge at temperature 0.1, and the average is taken as the final score.
• Overlap Ratio — range: [0, 1]
  • R_overlap = |H ∩ E^(M)| / |H ∪ E^(M)|, where H is the set of research issues identified by humans and E^(M) is the set identified by model M. Macro-averaged across papers.
• Human-only Ratio — range: [0, 1]
  • R_human-only = |H \ O^(M)| / |H|, measuring the proportion of human-identified issues missed by the model. Macro-averaged across papers.
• Model-only Ratio — range: [0, 1]
  • R_model-only = |E^(M) \ O^(M)| / |E^(M)|, measuring the proportion of model-identified issues not found by humans. Macro-averaged across papers.

Input / output format

Input: Full text of the academic paper to be reviewed.

Output: Structured peer review report containing scores (0-10) and textual explanations for Comprehensiveness, Specificity, Evidence Support, and Consistency.

Scoring recipe

def compute_metrics(predictions, gold):
    # LLM-as-Judge scoring (Sec 3.1)
    scores = []
    for _ in range(3):
        dim_scores = llm_judge.predict(predictions, temp=0.1)
        scores.append(mean(dim_scores))
    overall_quality = mean(scores)

    # Ratio metrics (Sec 3.2)
    overlap_ratios, human_only_ratios, model_only_ratios = [], [], []
    for paper in papers:
        H = gold[paper].issues
        E = predictions[paper].issues
        O = H.intersection(E)
        overlap_ratios.append(len(O) / len(H.union(E)))
        human_only_ratios.append(len(H - O) / len(H))
        model_only_ratios.append(len(E - O) / len(E))
    return overall_quality, mean(overlap_ratios), mean(human_only_ratios), mean(model_only_ratios)

Common pitfalls

• LLM-as-Judge scores are sensitive to temperature and prompt phrasing; the protocol strictly uses temp=0.1 and averages three independent runs.
• Weakness matching relies on GPT-5.2 to map surface-level statements to 'underlying research issues' via binary decisions, which may introduce semantic drift or over/under-matching.
• Pairwise win rates (Sec 3.3) require randomizing the presentation order of reviews to mitigate positional bias before feeding them to the judge LLM.

Evidence (verbatim from paper)

The judge model assigns a score from 0 to 10 for each dimension and computes the mean across all four dimensions as the Overall Quality score. To enhance evaluation stability and robustness, each review is independently scored three times under a temperature of 0.1, and the average is taken as the final score.

Citation

@misc{zhang2026echoreview,
  title={EchoReview: Learning Peer Review from the Echoes of Scientific Citations},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2602.00733}
}

• arXiv: 2602.00733