Survey Sum Eval

survey-sum-eval

Ask, Retrieve, Summarize: A Modular Pipeline for Scientific Literature Summarization — Achkar et al. (2025) (arXiv:2505.16349, 2025)

What this evaluates

Evaluates a modular retrieval-augmented generation pipeline for multi-document scientific literature summarization. It probes the model's ability to dynamically generate queries, retrieve relevant papers, and synthesize citation-aware, coherent summaries from multiple sources.

Datasets

• SurveySum — total 79; splits: test (79); repo https://github.com/unicamp-dl/surveysum

Metrics

• ROUGE-1 — range: [0, 1]
  • Recall-oriented n-gram overlap measuring unigram matching between generated and reference summaries.
• ROUGE-2 — range: [0, 1]
  • Recall-oriented n-gram overlap measuring bigram matching between generated and reference summaries.
• ROUGE-L — range: [0, 1]
  • Recall-oriented longest common subsequence matching at the sentence level.
• BERTScore — range: [0, 1]
  • Semantic similarity computed using contextual embeddings from PLMs (e.g., BERT) between generated and reference texts.
• Ref-F1 (primary) — range: [0, 1]
  • F1 score combining precision (proportion of correctly included references in the generated summary) and recall (proportion of ground-truth references captured).
• G-Eval — range: [1, 5]
  • Reference-free LLM-based evaluation using Chain-of-Thought reasoning to score criteria like coherence, coverage, fluency, and relevance on a 1-5 scale.
• CheckEval — range: [0, 1]
  • LLM-based checklist evaluation where the final score is the proportion of positive (yes) responses to structured sub-aspect questions.

Input / output format

Input: A collection of full-text scientific papers (average 7.38 per instance) retrieved via dynamic question generation, intended for multi-document summarization.

Output: A synthesized multi-document summary with inline citations aligned to academic standards and ground truth references.

Scoring recipe

def compute_ref_f1(pred_summary, gold_summary):
    pred_refs = extract_citations(pred_summary)
    gold_refs = extract_citations(gold_summary)
    tp = len(pred_refs.intersection(gold_refs))
    precision = tp / len(pred_refs) if pred_refs else 0.0
    recall = tp / len(gold_refs) if gold_refs else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1

Common pitfalls

• ROUGE-2 scores are inherently low for abstractive scientific summarization because pipelines prioritize semantic richness and paraphrasing over strict lexical bigram matching.
• LLM-based metrics (G-Eval, CheckEval) are sensitive to evaluator model choice and configuration, which can impact reproducibility and consistency.
• Using identical models for generation and evaluation introduces egocentric bias; the protocol explicitly separates them (GPT-4o-mini for generation, Phi-3-small for evaluation).

Evidence (verbatim from paper)

The evaluation employs a mix of traditional and LLM-based metrics to assess the quality of summaries in terms of content coverage, coherence, and citation alignment: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures the overlap between the generated summaries and the reference text. It calculates n-gram overlap, word sequence matching, and the longest common subsequences. Reference F1 Score (Ref-F1) measures how accurately the citations in the generated summaries align with those in the ground truth. It computes precision (proportion of correctly included references) and recall (proportion of ground-truth references captured in the generated summary), and combines them into an F1 score.

Citation

@misc{achkar2025askretrievesummarize,
  title={Ask, Retrieve, Summarize: A Modular Pipeline for Scientific Literature Summarization},
  author={Achkar et al. (2025)},
  year={2025},
  note={arXiv:2505.16349}
}

• arXiv: 2505.16349