Scizoom Eval

scizoom-eval

SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era — Jang et al. (2026) (arXiv:2603.16131, 2026)

What this evaluates

Evaluates a model's ability to perform hierarchical scientific summarization by generating three distinct granularity levels (Abstract, Key Contributions, TL;DR) from a single full-text input. It probes multi-granularity text compression and the model's capacity to maintain coherence across varying compression ratios within a single inference pass.

Datasets

• SciZoom — total 44946; splits: test (44946); repo https://github.com/janghana/SciZoom

Metrics

• unspecified summarization metric (primary) — range: other
  • Not explicitly defined in the provided text. Standard automatic summarization metrics (e.g., ROUGE, BERTScore) are typically used for this task.

Input / output format

Input: Full text of a scientific paper ($G_1$), with abstracts, bibliographies, and formatting artifacts removed.

Output: Three hierarchical summaries: Abstract ($G_2$), Key Contributions ($G_3$), and TL;DR ($G_4$).

Scoring recipe

def score(predictions, gold):
    scores = {}
    for level in ['abstract', 'contributions', 'tldr']:
        if level in gold and gold[level] is not None:
            scores[level] = compute_metric(predictions[level], gold[level])
        else:
            scores[level] = None
    return scores

Common pitfalls

• TL;DR ground truth is only available for 21,295 papers (47.4% of the corpus), meaning evaluation coverage is incomplete for this granularity level.
• Key Contributions for 66% of papers were synthetically generated via an LLM fallback pipeline, which may introduce noise or bias into the gold standard.
• The Pre/Post-LLM temporal split is based on submission deadlines (e.g., ICLR 2023 is Pre-LLM), not publication dates, which can cause temporal misalignment in era-based analysis.

Evidence (verbatim from paper)

For the TL;DR evaluation, we use the subset of 21,295 papers with author-provided ground truth, covering 47.4% of the corpus.

Citation

@misc{jang2026scizoom,
  title={SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era},
  author={Jang et al. (2026)},
  year={2026},
  note={arXiv:2603.16131}
}

• arXiv: 2603.16131