Author Centric Review Eval

author-centric-review-eval

Graph-Guided Passage Retrieval for Author-Centric Structured Feedback — Chitale et al. (2025) (arXiv:2505.14376, 2025)

What this evaluates

Evaluates an LLM's ability to generate structured, author-centric academic feedback (Summary, Strengths, Weaknesses, Questions) from long research papers. It probes the model's capacity to retrieve salient passages via graph-based retrieval and synthesize constructive pre-submission reviews without relying on full context or multi-agent systems.

Datasets

• ICLR 2024 (ICT) — total 5653; splits: train (-1), test (-1)
• CNT_10 — total 10; splits: test (10)

Metrics

• human evaluation (primary) — range: other
  • Human raters assess the generated Summary, Strengths, Weaknesses, and Questions sections against ground-truth reviews or predefined criteria for constructive feedback quality.

Input / output format

Input: Parsed academic paper represented as a hierarchical graph (nodes: headings, passages, sentences) or raw text document.

Output: Structured review containing four sections: Summary, Strengths, Weaknesses, and Questions.

Scoring recipe

def score_human_eval(predictions, gold_reviews):
    scores = []
    for pred, gold in zip(predictions, gold_reviews):
        scores.append(human_rater_score(pred, gold))
    return sum(scores) / len(scores)

Common pitfalls

• The evaluation explicitly excludes rating-based fields (Soundness, Presentation, Contribution, Confidence, Rating) to focus solely on constructive, author-centric feedback.
• Human evaluation is performed on a small supplementary set (CNT_10) alongside ICLR 2024 test data, so results may not generalize to larger unseen venues without caution.

Evidence (verbatim from paper)

Each paper is associated with multiple reviews containing sections such as Summary, Strengths, Weaknesses, Questions, Soundness, Presentation, Contribution, Confidence, and Rating. We retain only the Summary, Strengths, Weaknesses, and Questions sections, as these encode constructive feedback for authors-centric pre-submission support... To assess cross-venue performance, we curate a supplementary dataset of 10 papers from COLM and NeurIPS 2025 (CNT_10). This set complements the ICLR 2024 test data (ICT) for the human evaluation detailed in Section[6.3].

Citation

@misc{chitale2025graphguided,
  title={Graph-Guided Passage Retrieval for Author-Centric Structured Feedback},
  author={Chitale et al. (2025)},
  year={2025},
  note={arXiv:2505.14376}
}

• arXiv: 2505.14376