Billsum Eval

billsum-eval

BillSum: A Corpus for Automatic Summarization of US Legislation — Kornilova et al. (2019) (arXiv:1910.00523, 2019)

What this evaluates

This benchmark evaluates the ability of models to automatically generate concise and accurate summaries of complex, nested legislative texts. It probes extractive and abstractive summarization capabilities in a highly technical, domain-specific legal context.

Datasets

• BillSum — total 23455; splits: train (-1), test (-1); repo https://github.com/FiscalNote/BillSum

Metrics

• ROUGE F-Score (primary) — range: percent
  • Computes the F-measure (harmonic mean of precision and recall) for overlapping n-grams between the generated summary and the reference summary. Expressed as a percentage.

Input / output format

Input: Full text of a US Congressional or California state legislative bill.

Output: A generated summary of the bill, evaluated against a human-written reference summary.

Scoring recipe

def compute_rouge_f_score(generated, reference, n=2):
    gen_ngrams = ngrams(generated, n)
    ref_ngrams = ngrams(reference, n)
    overlap = len(gen_ngrams & ref_ngrams)
    precision = overlap / len(gen_ngrams) if gen_ngrams else 0
    recall = overlap / len(ref_ngrams) if ref_ngrams else 0
    if precision + recall == 0:
        return 0.0
    f_score = 2 * (precision * recall) / (precision + recall)
    return f_score * 100

Common pitfalls

• ROUGE F-scores are reported as percentages, not decimals.
• The dataset features nested, bulleted legislative structures and technical legal language, which can cause extractive models to struggle with sentence-level salience.
• Models trained on US Congressional bills often show performance drops when transferred to California state bills due to differences in summary structure and content.

Evidence (verbatim from paper)

The Rouge F-Score is used because it considers both the completeness and conciseness of the summary method.[11,12]

Citation

@misc{kornilova2019billsum,
  title={BillSum: A Corpus for Automatic Summarization of US Legislation},
  author={Kornilova et al. (2019)},
  year={2019},
  note={arXiv:1910.00523}
}

• arXiv: 1910.00523