Summarization Compression Eval

summarization-compression-eval

From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors — Zhou et al. (2026) (arXiv:2604.23277, 2026)

What this evaluates

This evaluation probes a model's ability to compress long documents into concise summaries while preserving topical coverage, cross-sentence coherence, and factual consistency under strict token budgets. It measures how well extractive or generative methods balance semantic relevance with structural discourse cues across diverse domains.

Datasets

• CNN/DailyMail — total ?; splits: test (-1)
• GovReport — total ?; splits: test (-1)
• arXiv — total ?; splits: test (-1)
• PubMed — total ?; splits: test (-1)

Metrics

• ROUGE-1 — range: [0, 1]
  • Computes unigram overlap between the generated summary and reference summaries using recall-based scoring.
• ROUGE-2 (primary) — range: [0, 1]
  • Computes bigram overlap between the generated summary and reference summaries, emphasizing phrase-level fluency and content matching.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence between generated and reference summaries to capture sentence-level structure.
• BERTScore — range: [0, 1]
  • Uses contextual embeddings from BERT to compute cosine similarity for precision, recall, and F1 between generated and reference tokens.
• QAFactEval — range: [0, 1]
  • Generates questions from the summary and checks if they can be answered from the source document, measuring factual consistency.

Input / output format

Input: A long-form document (news article, government report, arXiv paper, or PubMed article) and a specified token budget for the output summary.

Output: A set of selected sentences forming a summary that must not exceed the given token budget.

Scoring recipe

def compute_metrics(predictions, golds):
    # predictions and golds are lists of summary strings
    rouge = rouge_score(golds, predictions, use_stemmer=True)
    bs = bert_score(golds, predictions, lang='en')
    fe = [qafacteval(g, p) for g, p in zip(golds, predictions)]
    return {
        'rouge_1': rouge['rouge1'],
        'rouge_2': rouge['rouge2'],
        'rouge_l': rouge['rougeL'],
        'bertscore_f1': bs['f1'].mean(),
        'qafacteval': sum(fe) / len(fe)
    }

Common pitfalls

• Token budget constraints are strictly enforced; summaries exceeding the budget are invalid, yet some baselines ignore this during generation.
• ROUGE scores can be misleading for factual consistency; e.g., RankSum achieves higher ROUGE on CNN/DailyMail but lower QAFactEval, showing overlap does not guarantee faithfulness.
• Long-document datasets require modeling cross-sentence structure, not just local semantic similarity, or performance degrades significantly.

Evidence (verbatim from paper)

We evaluate summarization quality with ROUGE-1/2/L Lin (2004) and BERTScore (BS) Zhang et al. (2019), and assess factual consistency using QAFactEval (FE) Fabbri et al. (2022).

Citation

@misc{zhou2026from,
  title={From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors},
  author={Zhou et al. (2026)},
  year={2026},
  note={arXiv:2604.23277}
}

• arXiv: 2604.23277