Court Judgment Prediction Eval

court-judgment-prediction-eval

SemEval 2023 Task 6: LegalEval - Understanding Legal Texts — Modi et al. (2023) (arXiv:2304.09548, 2023)

What this evaluates

Tests a model's ability to predict binary case outcomes (accepted/denied) and generate human-readable explanations by citing relevant sentences from the input document. This probes joint reasoning, outcome forecasting, and justification generation in legal contexts.

Datasets

• LegalEval CJPE Dataset — total ?; splits: test (-1)

Metrics

• standard F1 score (primary) — range: [0, 1]
  • Standard F1 score for binary classification of case outcomes (accepted vs. denied).
• ROUGE-2 score — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation (ROUGE-2) measures the overlap of bigrams between the generated explanation and the gold explanation.

Input / output format

Input: A legal judgment document.

Output: A binary label (accepted/denied) and a set of explanatory sentences extracted from the document.

Scoring recipe

def compute_cjpe_metrics(pred_label, pred_explanation, gold_label, gold_explanation):
    tp = 1 if pred_label == gold_label == 'accepted' else 0
    fp = 1 if pred_label == 'accepted' and gold_label == 'denied' else 0
    fn = 1 if pred_label == 'denied' and gold_label == 'accepted' else 0
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    rouge2 = compute_rouge_n(pred_explanation, gold_explanation, n=2)
    return {'f1': f1, 'rouge2': rouge2}

Common pitfalls

• Explanation quality is hard to evaluate automatically; ROUGE-2 only measures n-gram overlap, not factual correctness or legal validity.
• Outcome prediction can be biased by dataset construction if not carefully debiased, leading to inflated scores on majority classes.

Evidence (verbatim from paper)

For the CJPE task, the evaluation for judgment prediction (binary classification) is done using the standard F1 score metric, and for the explanation sub-task, we use ROUGE-2 score

Citation

@misc{modi2023legaleval,
  title={SemEval 2023 Task 6: LegalEval - Understanding Legal Texts},
  author={Modi et al. (2023)},
  year={2023},
  note={arXiv:2304.09548}
}

• arXiv: 2304.09548