Xtragpt Eval

xtragpt-eval

XtraGPT: Context-Aware and Controllable Academic Paper Revision — Chen et al. (2025) (arXiv:2505.11336, 2025)

What this evaluates

Evaluates an LLM's ability to perform context-aware, instruction-guided revisions of academic paper sections. It probes controllable editing capabilities, specifically measuring adherence to revision instructions, clarity, conciseness, and alignment with scientific writing standards through automated pairwise comparisons and human scoring.

Datasets

• XtraQA — total 7040; splits: test (-1)

Metrics

• Length-controlled (LC) win rate (primary) — range: percent
  • Pairwise comparison between a model's revision and a baseline revision, judged by an LLM (alpaca_eval_gpt4_turbo_fn) using a length-controlled prompt to prevent verbosity bias. Calculated as the percentage of times the model's revision is preferred over the baseline.
• Human quality rating — range: other
  • Average score assigned by three human evaluators on a 1–5 scale across four dimensions: Instruction Following, Criteria Alignment, In-Context Reference, and Revision Acceptance.

Input / output format

Input: Original academic paper section text, a specific revision instruction/query, and optional context or reference material.

Output: Revised section text that addresses the instruction while maintaining argumentative coherence, clarity, and conciseness.

Scoring recipe

def compute_lc_win_rate(predictions, baselines, judge_model):
    wins = sum(1 for p, b in zip(predictions, baselines) if judge_model.compare(p, b, length_control=True) == 'pred')
    return wins / len(predictions) * 100

def compute_human_rating(predictions, evaluators):
    scores = [sum(e.score(p) for e in evaluators) / len(evaluators) for p in predictions]
    return sum(scores) / len(scores)

Common pitfalls

• LLM judges inherently favor longer outputs unless explicitly constrained with length-control prompts, which can artificially inflate win rates.
• Human evaluation scores show significant inter-rater variance (e.g., Revision Acceptance ranged from 2.46 to 4.01), requiring careful aggregation and multiple judges.
• The AI-Scientist automated evaluator was only validated on a small sample (54 papers), so its reliability for broader paper-level scoring may be limited.

Evidence (verbatim from paper)

Beyond the length-controlled win rate results presented in Table[4] and Table[6], we further assess the effectiveness of revisions through human evaluation; specifically, by measuring users’ willingness to adopt the generated revision in place of the original paragraph.

Citation

@misc{chen2025xtragpt,
  title={XtraGPT: Context-Aware and Controllable Academic Paper Revision},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2505.11336}
}

• arXiv: 2505.11336