Lexsumm Eval

lexsumm-eval

LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English — Santosh T.Y.S.S. et al. (2024) (arXiv:2410.09527, 2024)

What this evaluates

This benchmark evaluates the ability of sequence-to-sequence models to generate accurate, concise, and faithful summaries of long legal documents across multiple jurisdictions. It probes domain-specific summarization capabilities, testing how well models handle varying input lengths, compression ratios, and the balance between extractive and abstractive generation in legal English.

Datasets

• BillSum — total 22218; splits: train (16664), val (2222), test (3322)
• EurLexSum — total 1504; splits: train (1128), val (151), test (225)
• GovReport — total 19465; splits: train (14598), val (2919), test (1946)
• MultiLexSum-Long — total 4539; splits: train (3404), val (454), test (681)
• MultiLexSum-Short — total 3138; splits: train (2340), val (312), test (486)
• MultiLexSum-Tiny — total 1603; splits: train (1207), val (145), test (251)
• InAbs — total 7150; splits: train (5346), val (713), test (1069)
• UKAbs — total 793; splits: train (595), val (79), test (119)

Metrics

• Compression Ratio (primary) — range: ratio
  • The token ratio between the input document and the generated summary. Calculated as len(input_tokens) / len(summary_tokens).
• Coverage@1 — range: [0, 1]
  • Quantifies the extent to which a summary is derivative of the input text. It is the ratio of 1-grams in the summary that appear in an extractive fragment within the input.
• Density@1 — range: other
  • Quantifies how well the n-gram sequence of a summary can be described as a series of extractions. Defined as the average length of the extractive fragment to which each 1-gram in the summary belongs.
• Fusion score — range: other
  • Measures how summary sentences are synthesized from multiple sentences or compressed from a single sentence in the input. Computed as the standard deviation between the locations of output bigrams in the input.

Input / output format

Input: English legal documents (e.g., congressional bills, EU legislation, policy reports, civil rights lawsuits, or supreme court judgments) ranging from a few thousand to over 100,000 tokens.

Output: A human-written or model-generated summary (headnote) of the input document. Length varies by dataset granularity from a single sentence (Tiny) to multiple paragraphs (Long).

Scoring recipe

def evaluate(input_text, summary):
    in_tok = tokenize(input_text)
    out_tok = tokenize(summary)
    comp_ratio = len(in_tok) / len(out_tok)
    coverage_1 = count_ngrams_in_input(out_tok, n=1) / len(out_tok)
    density_1 = avg_extractive_fragment_length(out_tok, n=1)
    fusion = std_dev(bigram_locations_in_input(out_tok))
    return comp_ratio, coverage_1, density_1, fusion

Common pitfalls

• Confusing the three MultiLexSum granularities (Tiny, Short, Long), which demand drastically different output lengths and compression expectations.
• Assuming high Coverage@1 guarantees faithfulness, as dispersed extractive tokens can still be used abstractively, resulting in low Density scores.
• Overlooking jurisdiction-specific legal terminology and formatting differences across the eight datasets, which hinders zero-shot generalization.

Evidence (verbatim from paper)

(b) Compression Ratio Grusky et al. ([2018]) indicates the token ratio between the input to the summary. (c) Coverage@n Grusky et al. ([2018]) quantifies the extent to which a summary is derivative of a input text. It indicates the ratio of n-grams in the summary that are part of an extractive fragment within the input. (d) Density@n Grusky et al. ([2018]) quantifies how well the n-gram sequence of a summary can be described as a series of extractions.

Citation

@misc{santosh2024lexsumm,
  title={LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English},
  author={Santosh T.Y.S.S. et al. (2024)},
  year={2024},
  note={arXiv:2410.09527}
}

• arXiv: 2410.09527