Multilexsum Eval

multilexsum-eval

Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities — Shen et al. (2022) (arXiv:2206.10883, 2022)

What this evaluates

Evaluates abstractive and extractive summarization models on real-world civil rights lawsuits, testing their ability to synthesize information from extremely long multi-document sources and generate summaries at three distinct length granularities (long, short, tiny).

Datasets

• Multi-LexSum — total 4539; splits: train (3177), dev (454), test (908); repo https://github.com/multilexsum/dataset

Metrics

• ROUGE-1 F1 — range: [0, 1]
  • F1 score measuring unigram lexical overlap between the generated summary and the gold reference.
• ROUGE-2 F1 (primary) — range: [0, 1]
  • F1 score measuring bigram lexical overlap between the generated summary and the gold reference.
• ROUGE-L F1 — range: [0, 1]
  • F1 score based on the longest common subsequence between the generated summary and the gold reference.
• BERT Score F1 — range: [0, 1]
  • F1 score computed using cosine similarity between contextual sentence embeddings (DeBERTa) of the generated and reference texts.

Input / output format

Input: Source civil rights lawsuit documents (multi-document, up to 75,000 words). For progressive summarization tasks, the input also includes a ground-truth summary of a longer granularity.

Output: A single abstractive summary text at a specified granularity (long ~650 words, short ~130 words, or tiny ~25 words).

Scoring recipe

def evaluate(predictions, references):
    rouge_scores = compute_rouge(references, predictions, metrics=['rouge1', 'rouge2', 'rougeL'])
    bert_scores = compute_bert_score(references, predictions, model='DeBERTa')
    return {
        'ROUGE-1 F1': rouge_scores['rouge1'].fmeasure,
        'ROUGE-2 F1': rouge_scores['rouge2'].fmeasure,
        'ROUGE-L F1': rouge_scores['rougeL'].fmeasure,
        'BERT Score F1': bert_scores.fmeasure.mean()
    }

Common pitfalls

• Models consistently under-generate long summaries, failing to match the ~650-word ground truth length (best average was ~416 words).
• Pipeline approaches that feed generated summaries into subsequent models degrade performance significantly compared to using gold summaries as input.
• Extractive baselines and sentence-embedding models perform worse than abstractive models, especially for shorter, more abstractive targets.

Evidence (verbatim from paper)

Automatic Evaluation Rouge-{1,2,L} [37] and BERT Score (BS) [61] are used to compute the lexical and estimated semantic overlap between the generated and gold summaries. We use the DeBERTA [24] model for sentence embedding following the authors' suggestion. We report the average of F1 measures for Rouge and BS on the test set, and include the number of generated words for reference.

Citation

@misc{shen2022multilexsum,
  title={Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities},
  author={Shen et al. (2022)},
  year={2022},
  note={arXiv:2206.10883}
}

• arXiv: 2206.10883