Classactionprediction Eval

classactionprediction-eval

ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US — Semo et al. (2022) (arXiv:2211.00582, 2022)

What this evaluates

Predicts the outcome (win or lose) of U.S. class action lawsuits based on plaintiff complaint texts. It probes a model's ability to extract legally relevant allegations from long-form, unverified legal documents and make binary judgment predictions.

Datasets

• ClassActionPrediction — total 10800; splits: 5-fold cross-validation (-1); repo https://github.com/darrow-labs/ClassActionPrediction

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted win/lose labels out of total cases. Reported as mean ± standard deviation across 5 random seeds/folds.

Input / output format

Input: Plaintiff complaint text (full text, unified allegations, or separated allegations), truncated to a maximum sequence length (2,048 tokens for Longformer/BigBird, 512 tokens for other models).

Output: Binary label: 'win' or 'lose'.

Scoring recipe

correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)
# Report mean and standard deviation of accuracy across 5 cross-validation folds/seeds

Common pitfalls

• Truncating full complaint texts to 2,048 tokens often cuts off the allegations section (which appears later in the document), making the 'Full Text' variant artificially harder than the allegations-only variants.
• Legal-domain pretrained models show a smaller performance gain over BERT on this dataset compared to other benchmarks because the complaint data was unseen during pretraining, unlike the case law used in other tasks.

Evidence (verbatim from paper)

For all experiments, we truncated the text to the model's maximum sequence length (2,048 for Longformer and BigBird, 512 otherwise), unless otherwise specified. All experiments have been performed on the binarized labels (win or lose). We ran the experiments with 5-fold cross-validation and averaged across 5 random seeds. Results are reported in the mean±std format averaged accuracy across 5 random seeds.

Citation

@misc{semo2022classactionprediction,
  title={ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US},
  author={Semo et al. (2022)},
  year={2022},
  note={arXiv:2211.00582}
}

• arXiv: 2211.00582