Casesumm Eval

casesumm-eval

CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions — Heddaya et al. (2024) (arXiv:2501.00097, 2024)

What this evaluates

Evaluates the ability of language models to generate accurate, concise, and legally faithful summaries of long U.S. Supreme Court opinions. It probes the alignment between automatic NLP metrics and expert human judgment in a high-stakes, domain-specific summarization task.

Datasets

• CaseSumm — total 25600; splits: test (-1)

Metrics

• ROUGE (primary) — range: [0, 1]
  • Computes n-gram overlap between generated summary and reference syllabus. The paper reports both recall (sensitivity) and precision (specificity) scores.
• BERTScore — range: [0, 1]
  • Measures semantic similarity using contextual embeddings from BERT, reporting recall and precision metrics.
• G-Eval — range: [1, 5]
  • LLM-based evaluation scoring summaries on dimensions including Coherence, Relevance, Fluency, Sensitivity, Specificity, Clarity, and Style using adapted prompts.

Input / output format

Input: Full text of a U.S. Supreme Court opinion.

Output: A generated summary of the opinion.

Scoring recipe

def compute_metrics(summary, reference):
    rouge_r = rouge_score(reference, summary, rouge_type='ROUGE-L', average='recall')
    bert_r = bertscore_score(reference, summary, metric='recall')
    g_eval_scores = llm_evaluate(summary, reference, dimensions=['Coherence', 'Relevance', 'Fluency', 'Sensitivity', 'Specificity', 'Clarity', 'Style'])
    human_ranking = expert_judges_rank(summary, reference)
    return rouge_r, bert_r, g_eval_scores, human_ranking

Common pitfalls

• Automatic metrics like ROUGE and BERTScore heavily favor models that maximize recall, often at the cost of brevity and readability, which contradicts expert human preference.
• High automatic scores do not guarantee factual accuracy or legal faithfulness; models can achieve strong metric scores while containing hallucinations or precedent misattributions.
• Reference syllabuses are typically longer than desired summaries, causing automatic metrics to penalize concise, human-preferred outputs.

Evidence (verbatim from paper)

Overall, we find that fine-tuning Mistral is particularly effective at improving the recall scores across all the metrics: ROUGE recall scores increase by an average of 21 points, BERTScore recall by 15 points. However, effects of fine-tuning on precision are weaker and more mixed. Perhaps fine-tuning sacrifices brevity for inclusion of more words in a syllabus, i.e., improves the sensitivity of summaries at a cost to specificity.

Citation

@misc{heddaya2024casesumm,
  title={CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions},
  author={Heddaya et al. (2024)},
  year={2024},
  note={arXiv:2501.00097}
}

• arXiv: 2501.00097