# Lexsumm Eval

> This benchmark evaluates the ability of sequence-to-sequence models to generate accurate, concise, and faithful summaries of long legal documents across multiple jurisdictions. It probes domain-specific summarization capabilities, testing how well models handle varying input lengths, compression ratios, and the balance between extractive and abstractive generation in legal English. Use when the user wants to benchmark on BillSum, EurLexSum, GovReport, MultiLexSum-Long, MultiLexSum-Short, MultiLexSum-Tiny, InAbs, UKAbs, or asks about evaluating this task. Reports Compression Ratio.

- Skill: `qhjqhj00/lexsumm-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/lexsumm-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/lexsumm-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/lexsumm-eval

---


# lexsumm-eval

> LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English — Santosh T.Y.S.S. et al. (2024) (arXiv:2410.09527, 2024)

## What this evaluates

This benchmark evaluates the ability of sequence-to-sequence models to generate accurate, concise, and faithful summaries of long legal documents across multiple jurisdictions. It probes domain-specific summarization capabilities, testing how well models handle varying input lengths, compression ratios, and the balance between extractive and abstractive generation in legal English.

## Datasets

- **BillSum** — total 22218; splits: train (16664), val (2222), test (3322)
- **EurLexSum** — total 1504; splits: train (1128), val (151), test (225)
- **GovReport** — total 19465; splits: train (14598), val (2919), test (1946)
- **MultiLexSum-Long** — total 4539; splits: train (3404), val (454), test (681)
- **MultiLexSum-Short** — total 3138; splits: train (2340), val (312), test (486)
- **MultiLexSum-Tiny** — total 1603; splits: train (1207), val (145), test (251)
- **InAbs** — total 7150; splits: train (5346), val (713), test (1069)
- **UKAbs** — total 793; splits: train (595), val (79), test (119)

## Metrics

- `Compression Ratio` **(primary)** — range: ratio
  - The token ratio between the input document and the generated summary. Calculated as len(input_tokens) / len(summary_tokens).
- `Coverage@1` — range: [0, 1]
  - Quantifies the extent to which a summary is derivative of the input text. It is the ratio of 1-grams in the summary that appear in an extractive fragment within the input.
- `Density@1` — range: other
  - Quantifies how well the n-gram sequence of a summary can be described as a series of extractions. Defined as the average length of the extractive fragment to which each 1-gram in the summary belongs.
- `Fusion score` — range: other
  - Measures how summary sentences are synthesized from multiple sentences or compressed from a single sentence in the input. Computed as the standard deviation between the locations of output bigrams in the input.

## Input / output format

**Input**: English legal documents (e.g., congressional bills, EU legislation, policy reports, civil rights lawsuits, or supreme court judgments) ranging from a few thousand to over 100,000 tokens.

**Output**: A human-written or model-generated summary (headnote) of the input document. Length varies by dataset granularity from a single sentence (Tiny) to multiple paragraphs (Long).

## Scoring recipe

```python
def evaluate(input_text, summary):
    in_tok = tokenize(input_text)
    out_tok = tokenize(summary)
    comp_ratio = len(in_tok) / len(out_tok)
    coverage_1 = count_ngrams_in_input(out_tok, n=1) / len(out_tok)
    density_1 = avg_extractive_fragment_length(out_tok, n=1)
    fusion = std_dev(bigram_locations_in_input(out_tok))
    return comp_ratio, coverage_1, density_1, fusion
```

## Common pitfalls

- Confusing the three MultiLexSum granularities (Tiny, Short, Long), which demand drastically different output lengths and compression expectations.
- Assuming high Coverage@1 guarantees faithfulness, as dispersed extractive tokens can still be used abstractively, resulting in low Density scores.
- Overlooking jurisdiction-specific legal terminology and formatting differences across the eight datasets, which hinders zero-shot generalization.

## Evidence (verbatim from paper)

> (b) Compression Ratio Grusky et al. ([2018]) indicates the token ratio between the input to the summary. (c) Coverage@n Grusky et al. ([2018]) quantifies the extent to which a summary is derivative of a input text. It indicates the ratio of n-grams in the summary that are part of an extractive fragment within the input. (d) Density@n Grusky et al. ([2018]) quantifies how well the n-gram sequence of a summary can be described as a series of extractions.

## Citation

```bibtex
@misc{santosh2024lexsumm,
  title={LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English},
  author={Santosh T.Y.S.S. et al. (2024)},
  year={2024},
  note={arXiv:2410.09527}
}
```

- arXiv: 2410.09527