Legalbenchpt Eval

legalbenchpt-eval

LegalBench.PT: A Benchmark for Portuguese Law — Canaverde et al. (2025) (arXiv:2502.16357, 2025)

What this evaluates

Evaluates large language models' ability to reason about and classify Portuguese legal concepts across 31 distinct legal domains. It probes zero-shot question-answering capabilities using multiple-choice, true/false, matching, and case-analysis formats derived from law exam questions.

Datasets

• LegalBench.PT — total ?; splits: test (-1)

Metrics

• balanced accuracy (primary) — range: [0, 1]
  • Average of recall scores across all classes (answer options). For binary true/false tasks, it equals standard accuracy.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall, used for multiple-selection and matching tasks where answers are sets of options.
• Jaccard similarity — range: [0, 1]
  • Size of intersection divided by size of union between predicted and gold option sets, used to compute agreement between human annotators.

Input / output format

Input: A prompt specifying the legal area, detailed instructions on the question type (e.g., whether there is one or multiple correct answers), and the expected response format.

Output: Extracted letter options, true/false classifications, and matching pairs according to the instructed format.

Scoring recipe

def compute_metric(predictions, golds, q_type):
    if q_type in ['mc', 'cloze', 'case', 'tf']:
        recalls = []
        for opt in options:
            correct = sum(1 for p, g in zip(predictions, golds) if p == opt and g == opt)
            total_gold = sum(1 for g in golds if g == opt)
            recalls.append(correct / total_gold if total_gold > 0 else 0)
        return sum(recalls) / len(recalls)
    elif q_type in ['multi_select', 'matching']:
        preds_set = set(predictions)
        golds_set = set(golds)
        tp = len(preds_set & golds_set)
        fp = len(preds_set - golds_set)
        fn = len(golds_set - preds_set)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Ambiguity in legal questions: ~14.8% of questions may have multiple valid interpretations, making strict gold-standard matching unfair.
• Small question sets in niche fields (e.g., Maritime, Insolvency Law) yield less reliable performance estimates and do not represent comprehensive evaluations.
• Potential generation bias: The dataset was created using GPT-4o, which may artificially inflate its scores compared to other models.

Evidence (verbatim from paper)

We evaluate multiple-choice, cloze tasks, case analysis, and true/false questions separately using balanced accuracy. For multiple selection and matching questions, we use the $F_{1}$ score. We aggregate the results from the different quesion types and fields of law using a weighted average.

Citation

@misc{canaverde2025legalbenchpt,
  title={LegalBench.PT: A Benchmark for Portuguese Law},
  author={Canaverde et al. (2025)},
  year={2025},
  note={arXiv:2502.16357}
}

• arXiv: 2502.16357