Apres Paper Revision Eval

apres-paper-revision-eval

APRES: An Agentic Paper Revision and Evaluation System — Zhao et al. (2026) (arXiv:2603.03142, 2026)

What this evaluates

This protocol evaluates an LLM's ability to predict a paper's future scientific impact based on its text and peer reviews, and its ability to iteratively revise the manuscript to maximize that predicted impact. It probes the model's capacity for rubric discovery, agentic text editing, and alignment with human expert preferences.

Datasets

• ICLR & NeurIPS Peer Review Dataset — total 26707; splits: train (-1), val (-1), test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error between predicted and actual influential citation counts from Semantic Scholar. Formula: MAE = (1/N) * sum(|y_true - y_pred|).
• Improvement Score ($\Delta S$) (primary) — range: other
  • Change in predicted impact score after revision, calculated as $\Delta S = S_{rev} - S_{ori}$, where scores are generated by the discovered negative binomial regression rubric.
• Human Preference Rate — range: percent
  • Percentage of paper pairs where the majority of three PhD-level annotators preferred the AI-revised version over the original in a blind pairwise comparison.

Input / output format

Input: Original paper text, experimental results (protected), and the discovered evaluation rubric.

Output: A series of diff-based search/replace blocks specifying exact text edits to improve the manuscript.

Scoring recipe

def compute_mae(predictions, gold_citations):
    return sum(abs(p - g) for p, g in zip(predictions, gold_citations)) / len(predictions)

def compute_delta_s(original_score, revised_score):
    return revised_score - original_score

def compute_human_preference_rate(pairs, annotator_votes):
    majority_wins = sum(all(votes) or sum(votes) >= 2 for votes in annotator_votes)
    return (majority_wins / len(pairs)) * 100

Common pitfalls

• Raw human reviewer scores are poor predictors of future citations (MAE ~5.0), so using them as a baseline or proxy for impact is misleading.
• The revision agent only modifies presentation, clarity, and argumentation; it cannot improve core methodological performance or experimental results, which are explicitly protected from editing.
• The dataset's acceptance rate is artificially inflated because NeurIPS only makes a subset of rejected papers public, skewing the train/val/test distribution.

Evidence (verbatim from paper)

The first metric is the Improvement Score ($\Delta S$), which is the change between the final and original predicted impact scores, $\Delta S=S_{\text{rev}}-S_{\text{ori}}$. The impact scores are predicted by the discovered rubric in Sec. [3.1] as they correlate the most with the actual citation numbers. A positive $\Delta S$ provides quantitative evidence that the LLM-generated feedback was sufficiently constructive and actionable to guide the agent to produce a verifiably better paper.

Citation

@misc{zhao2026apres,
  title={APRES: An Agentic Paper Revision and Evaluation System},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.03142}
}

• arXiv: 2603.03142