Academiceval Eval

academiceval-eval

Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems — Tao Feng et al. (2026) (arXiv:2604.12231, 2026)

What this evaluates

Evaluates an LLM's ability to perform long-context summarization and synthesize related work sections by retrieving and reasoning over heterogeneous academic memory chunks.

Datasets

• AcademicEval-abstract — total ?; splits: test (-1); repo https://github.com/ulab-uiuc/Thought-Retriever
• AcademicEval-related — total ?; splits: test (-1); repo https://github.com/ulab-uiuc/Thought-Retriever

Metrics

• F1 score (primary) — range: [0, 1]
  • Token-level F1 score computed between the model's generated text and the gold reference (original abstract, expert LLM summary, or original related work).
• Win rate — range: percent
  • Pairwise comparison metric indicating the percentage of instances where the model's output is preferred over a baseline's output.

Input / output format

Input: For Abstract-single/multi: academic paper text with abstract and conclusion sections removed. For Related-multi: target paper title and abstract, plus a set of memory chunks containing abstracts of other papers (some cited, some random).

Output: Generated abstract (for single/multi tasks) or generated related work section (for related task).

Scoring recipe

def score(generated, gold):
    gen_tokens = set(generated.split())
    gold_tokens = set(gold.split())
    precision = len(gen_tokens & gold_tokens) / len(gen_tokens) if gen_tokens else 0
    recall = len(gen_tokens & gold_tokens) / len(gold_tokens) if gold_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• The gold label for the multi-paper abstract task is generated by an expert LLM rather than human annotators, which may propagate model biases.
• Evaluation conflates retrieval quality and generation quality in the Related-multi task, making it hard to isolate whether failures stem from chunk selection or text synthesis.

Evidence (verbatim from paper)

Experiments on AcademicEval show a 7.6% F1 gain and 16% win rate over baselines... The LLM is presented with one or more papers with the abstract and conclusion sections removed and is tasked with writing an abstract. For Abstract-single, the generated abstract is directly compared with the paper’s original abstract. For Abstract-multi, the generated abstract is compared with a summary of abstracts from all the provided papers, which is generated by an expert LLM as a label.

Citation

@misc{feng2026thoughtretriever,
  title={Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems},
  author={Tao Feng et al. (2026)},
  year={2026},
  note={arXiv:2604.12231}
}

• arXiv: 2604.12231