Maud Eval

maud-eval

MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding — Wang et al. (2023) (arXiv:2301.00876, 2023)

What this evaluates

This benchmark evaluates a model's ability to perform legal reading comprehension on merger agreements by answering specialized deal point questions. It probes the model's capacity to interpret complex contractual clauses and handle imbalanced classification tasks across various legal categories.

Datasets

• MAUD — total 39000; splits: train (-1), dev (-1), test (-1); repo http://github.com/TheAtticusProject/maud

Metrics

• AUPR (primary) — range: [0, 1]
  • Area under the precision-recall curve, averaged across different questions and answers to account for imbalanced answer distributions.

Input / output format

Input: A merger agreement text excerpt paired with a specific deal point question.

Output: A predicted class label for the deal point question.

Scoring recipe

def compute_aupr(y_true, y_prob):
    precisions, recalls, _ = precision_recall_curve(y_true, y_prob)
    return auc(recalls, precisions)
# Average AUPR across all deal point questions and answer types as detailed in Appendix A.5

Common pitfalls

• Imbalanced answer distributions make standard accuracy or F1 misleading; AUPR is required.
• Over 50% of texts exceed 512 tokens, so truncation or long-context models (e.g., BigBird) are necessary for fair evaluation.
• Single-task models typically outperform multi-task models by ~4 pp AUPR, contrary to typical multi-task benefits.

Evidence (verbatim from paper)

Many MAUD questions have an imbalanced answer distribution, so we use area under the precision-recall curve (AUPR) as our primary metric. See Appendix A.5 for details on how we average AUPR across different questions and answers.

Citation

@misc{wang2023maud,
  title={MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2301.00876}
}

• arXiv: 2301.00876