Beyond Rating Eval

beyond-rating-eval

Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews — Li et al. (2026) (arXiv:2604.19502, 2026)

What this evaluates

Evaluates AI-generated scientific paper reviews across five dimensions: content faithfulness, argumentative alignment, focus consistency, question constructiveness, and AI-likeness. It probes whether models can replicate human-like evaluative reasoning and scoring rather than merely predicting scalar ratings.

Datasets

• AI Review Benchmark — total ?; splits: test (-1)

Metrics

• MAE (primary) — range: [0, 10]
  • Mean Absolute Error between the model's predicted rating and the human expert's rating. Calculated as the average of absolute differences across all reviewed papers.
• Recall/Precision/F1 — range: [0, 1]
  • Standard information retrieval metrics applied to atomic claim extraction for Strength and Weakness sections. Recall measures overlap with human gold-standard claims, Precision measures exactness of model-generated claims, and F1 is their harmonic mean.
• KL Divergence — range: [0, ∞)
  • Kullback-Leibler divergence between the probability distribution of claim categories (e.g., Novelty, Soundness, Experiments) in the model's review versus the human expert's review. Lower values indicate better perspective alignment.
• Binocular Score — range: [0, ∞)
  • A perplexity-derived metric from the Binoculars detection model that quantifies textual quality and cognitive depth. Higher scores indicate substantive reasoning and deviation from formulaic AI patterns.
• Question Score (QS) — range: percent
  • A quality assessment metric for the Questions section of the review. Higher scores indicate more constructive and well-formed questions.

Input / output format

Input: Scientific paper manuscript. Models receive the full text and are prompted to generate a structured peer review.

Output: Structured review containing Summary, Strengths, Weaknesses, Questions, and a scalar Rating score.

Scoring recipe

def evaluate(predictions, gold):
    # 1. Rating MAE
    mae = mean(abs(p['rating'] - g['rating']) for p, g in zip(predictions, gold))
    
    # 2. Strength/Weakness Alignment (Recall/Precision/F1)
    tp = sum(1 for p in predictions if p['weakness'] in gold[0]['weakness'])
    fn = sum(1 for g in gold[0]['weakness'] if g not in predictions[0]['weakness'])
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    precision = tp / (tp + len(predictions[0]['weakness'])) if predictions[0]['weakness'] else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    
    # 3. KL Divergence (claim category distribution vs human)
    kl = compute_kl_divergence(model_claim_dist, human_claim_dist)
    
    # 4. Binocular Score (AI-likeness/perplexity)
    bs = binocular_model.predict(predictions)
    
    return {'MAE': mae, 'Recall': recall, 'Precision': precision, 'F1': f1, 'KL': kl, 'BS': bs}

Common pitfalls

• High embedding-based summary scores often reflect AI verbosity and detail retention rather than true synthesis, creating algorithmic bias against human-like brevity.
• Precision in Strength/Weakness fields is uniformly low across models due to brief outputs, making it an insufficient metric for differentiating performance.
• Low MAE does not guarantee review quality; models can superficially fit human score distributions via simple prompting without achieving semantic depth or evaluative utility.

Evidence (verbatim from paper)

Metrics. As shown in Section 4, we employ a comprehensive suite of metrics to assess different aspects of the system. Strength and Weakness evaluations are conducted via Recall, Precision, F1, and KL divergence. For Question analysis, we monitor Question Score (QS.) and KL divergence, while AI-likelyhood detection relies on AI rate and the Binocular Score (BS.). Finally, predictive accuracy in rating evaluation is measured using MAE.

Citation

@misc{li2026beyondrating,
  title={Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.19502}
}

• arXiv: 2604.19502