Squality Eval

squality-eval

SQuALITY: Building a Long-Document Summarization Dataset the Hard Way — Wang et al. (2022) (arXiv:2205.11465, 2022)

What this evaluates

Evaluates long-document, question-focused summarization quality through structured human ratings and automatic metric correlation. It probes a model's ability to generate accurate, comprehensive, and high-quality summaries that align with human preferences rather than relying on surface-level n-gram overlap.

Datasets

• SQuALITY — total ?; splits: test (-1); repo https://github.com/nyu-mll/SQuALITY

Metrics

• Human Rating (1-100) (primary) — range: [0, 100]
  • Annotators rate each summary on a 1-100 scale for correctness, coverage, and overall quality. Ratings are averaged across three annotators per instance.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap between generated summary and reference(s), calculated as F1 score.

Input / output format

Input: Source story text and a specific question to be answered by the summary.

Output: A generated summary text answering the question.

Scoring recipe

def score_human(summary, gold):
    ratings = []
    for annotator in annotators:
        for prop in ['correctness', 'coverage', 'overall_quality']:
            ratings.append(annotator.rate(summary, prop, scale=100))
    return sum(ratings) / len(ratings)

def score_rouge1(summary, references):
    return compute_rouge_n(summary, references, n=1).fmeasure

Common pitfalls

• Relying exclusively on automatic metrics like ROUGE or BERTScore, as they show near-zero or negative correlation with human judgments when evaluating only model outputs.
• Assuming that using multiple references improves automatic metric correlation with human quality judgments; the paper shows metric values are similar across references, so taking the max or average does not substantially change correlation.
• Failing to compensate annotators for reading the source story if they haven't previously read it, which can bias ratings.

Evidence (verbatim from paper)

For each response, we ask workers to rate the response for three properties: correctness, coverage, and overall quality. ... the response is rated on a scale from 1-100 ... We annotate 20 stories (100 questions) this way, with three Upwork workers completing each task. For each property, we average the ratings across annotators. ... we argue that existing automatic evaluation metrics such as ROUGE do not adequately reflect the differences between model- and human-written summaries.

Citation

@misc{wang2022squality,
  title={SQuALITY: Building a Long-Document Summarization Dataset the Hard Way},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2205.11465}
}

• arXiv: 2205.11465