Scrolls Eval

scrolls-eval

Efficient Long-Text Understanding with Short-Text Models — Ivgi et al. (2022) (arXiv:2208.00748, 2022)

What this evaluates

Evaluates long-text understanding capabilities across summarization, question answering, and natural language inference tasks. It probes whether models can effectively process and extract information from documents exceeding standard context windows (up to 16K tokens) using chunked encoding and cross-chunk fusion.

Datasets

• SCROLLS — total ?; splits: dev (-1), test (-1)

Metrics

• Rouge-1/Rouge-2/Rouge-L — range: [0, 1]
  • Standard ROUGE metrics measuring n-gram overlap between generated summaries and reference summaries. Calculated separately for ROUGE-1, ROUGE-2, and ROUGE-L.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for open-ended QA tasks (Qasper, NarrativeQA). Measures overlap between predicted answer spans and gold spans.
• Exact Match (EM) — range: [0, 1]
  • Binary score indicating whether the predicted answer exactly matches the gold answer (for QuALITY) or the predicted NLI label matches the gold label (for ContractNLI).
• Avg SCROLLS score (primary) — range: [0, 1]
  • Official benchmark aggregation metric that averages normalized performance across all seven SCROLLS sub-datasets according to the SCROLLS leaderboard protocol.

Input / output format

Input: Long document (up to 16K tokens) partitioned into overlapping chunks of size 256 with 50% padding. For QA and NLI tasks, the question or hypothesis is prepended as a prefix to the document.

Output: Generated text corresponding to the task: summary for summarization datasets, answer span or option for QA datasets, and class label (entailment/contradiction/neutral) for NLI.

Scoring recipe

def evaluate(predictions, golds, task_type):
    if task_type in ['summarization']:
        return rouge1, rouge2, rouge_l(predictions, golds)
    elif task_type in ['qa_open', 'nli']:
        return f1_score(predictions, golds)
    elif task_type == 'qa_mc':
        return exact_match(predictions, golds)
    return average_of_all_task_scores(predictions, golds)

Common pitfalls

• QuALITY performance is dominated by model scale and commonsense knowledge rather than long-context retrieval, so long-context methods show negligible gains compared to baselines.
• Using non-overlapping chunks causes significant performance drops (≥10% relative gain loss) due to information loss at chunk boundaries.
• The 'Avg' SCROLLS score is a task-specific aggregation defined by the benchmark, not a simple arithmetic mean of all reported metrics.

Evidence (verbatim from paper)

For each task, we use the official evaluation metrics defined in SCROLLS, which are based on the metrics from the original datasets. All summarization datasets (GovReport, SummScreenFD and QMSum) show impressive gains of up to 35% compared to their baseline scores, across all metrics (Rouge-1/Rouge-2/Rouge-L (Lin, 2004)) and for all three backbone models.

Citation

@misc{ivgi2022efficientlongtext,
  title={Efficient Long-Text Understanding with Short-Text Models},
  author={Ivgi et al. (2022)},
  year={2022},
  note={arXiv:2208.00748}
}

• arXiv: 2208.00748