Cuad Eval

cuad-eval

CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review — Hendrycks et al. (2021) (arXiv:2103.06268, 2021)

What this evaluates

Evaluates a model's ability to identify and extract relevant text spans from legal contracts corresponding to specific clause categories. It probes domain-specific information extraction and needle-in-a-haystack detection under severe class imbalance.

Datasets

• CUAD — total 500; splits: train (-1), test (-1)

Metrics

• AUPR — range: percent
  • Area under the precision-recall curve computed by sweeping the model's confidence threshold across all predictions.
• Precision@80% Recall (primary) — range: percent
  • Precision achieved when the confidence threshold is tuned to yield exactly 80% recall.
• Precision@90% Recall — range: percent
  • Precision achieved when the confidence threshold is tuned to yield exactly 90% recall.

Input / output format

Input: A legal contract document (context) and a target label category (question) with a short description. The model receives the text and predicts start/end token positions for relevant spans.

Output: Predicted start and end token positions for each label category, accompanied by a confidence probability.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    # 1. Normalize strings: lowercase, remove punctuation, split by space
    # 2. For each prediction span, compute Jaccard similarity J(A,B) vs each GT span
    # 3. Match prediction to GT if max J(A,B) >= 0.5; else mark as False Positive
    # 4. Unmatched GT spans are False Negatives; matched are True Positives
    # 5. Vary confidence threshold to compute Precision and Recall at each step
    # 6. Compute AUPR by integrating the PR curve
    # 7. Extract Precision at 80% and 90% recall from the curve
    return AUPR, Precision_at_80, Precision_at_90

Common pitfalls

• Severe class imbalance (>99% of sliding windows are negative) can cause models to trivially predict empty spans unless negative samples are downweighted during training.
• Span matching uses a Jaccard similarity threshold of 0.5 on normalized word sets, which may penalize minor tokenization or punctuation differences.
• Precision@X% Recall requires precise thresholding of confidence scores; curve interpolation or threshold selection methods can affect reported values.

Evidence (verbatim from paper)

We use the Area Under the Precision-Recall curve (AUPR) and Precision at 80% and 90% Recall as our primary metrics. ... We determine whether a highlighted text span matches the ground truth with the Jaccard similarity coefficient. ... We use the threshold 0.5≤J(A,B) for determining matches.

Citation

@misc{hendrycks2021cuad,
  title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
  author={Hendrycks et al. (2021)},
  year={2021},
  note={arXiv:2103.06268}
}

• arXiv: 2103.06268