Caselawqa Eval

caselawqa-eval

Lawma: The Power of Specialization for Legal Annotation — Ricardo Dominguez-Olmedo et al. (2024) (arXiv:2407.16615, 2024)

What this evaluates

This benchmark probes a model's ability to perform fine-grained legal text classification and annotation. It tests whether models can accurately extract specific legal features, such as precedent alteration, issue areas, or ideological valence, from lengthy court opinions using multiple-choice prompts.

Datasets

• CaselawQA — total 718971; splits: train (-1), val (-1), test (143635); repo https://github.com/socialfoundations/lawma

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of examples. Aggregate performance is computed as a weighted average across tasks, where each task's accuracy is weighted proportional to its number of examples.

Input / output format

Input: A prompt template containing a general task description, the full text of a U.S. court opinion, a multiple-choice question about a specific legal feature, and answer options labeled A, B, etc.

Output: The model must generate a step-by-step reasoning chain followed by the exact phrase "The final answer is [final_answer]", where [final_answer] is a single uppercase letter (A-Z) or a numerical value.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if "The final answer is" in pred:
            extracted = pred.split("The final answer is")[-1].strip().split()[0]
        else:
            extracted = pred.strip().split()[-1]
        if extracted == gold:
            correct += 1
    return correct / len(gold_labels)

Common pitfalls

• Class imbalance is mitigated by subsampling the majority class so it never exceeds twice the size of all other classes combined, meaning a constant majority-class predictor caps at 50% accuracy.
• Models are prompted with chain-of-thought instructions, which significantly increases inference latency and token usage compared to direct multiple-choice answering.
• Tasks vary drastically in complexity and number of answer choices (up to 10+ classes), so aggregate accuracy is a weighted average proportional to task example counts, not a simple mean across tasks.

Evidence (verbatim from paper)

We use accuracy as the evaluation metric. Since the tasks we consider involve vastly differing numbers of answer choices, accuracy provides an interpretable and comparable measure of performance. Additionally, accuracy is the standard metric used in knowledge-testing LLM benchmarks. For completeness, we also report balanced accuracy and macro-averaged F1 score in Appendix[B]. When reporting aggregate performance across multiple tasks (e.g., all Supreme Court tasks), we compute the average accuracy across all task examples. This amounts to a weighted average where tasks are weighted proportional to the number of task examples available.

Citation

@misc{dominguezolmedo2024lawma,
  title={Lawma: The Power of Specialization for Legal Annotation},
  author={Ricardo Dominguez-Olmedo et al. (2024)},
  year={2024},
  note={arXiv:2407.16615}
}

• arXiv: 2407.16615