Rhetorical Roles Eval

rhetorical-roles-eval

SemEval 2023 Task 6: LegalEval - Understanding Legal Texts — Modi et al. (2023) (arXiv:2304.09548, 2023)

What this evaluates

Probes a model's ability to segment long, unstructured legal documents into semantically coherent units and assign each sentence a specific rhetorical role label (e.g., Facts, Ratio, Arguments). This capability is fundamental for downstream legal AI applications like summarization and precedent search.

Datasets

• LegalEval RR Dataset — total ?; splits: test (-1)

Metrics

• weighted F1 score (primary) — range: [0, 1]
  • Weighted average of per-class F1 scores, where weights correspond to the support (number of true instances) for each class.

Input / output format

Input: A legal judgment document represented as a sequence of sentences.

Output: A sequence of rhetorical role labels, one for each input sentence.

Scoring recipe

def compute_weighted_f1(predictions, gold, classes):
    scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        scores.append(f1 * sum(1 for g in gold if g == cls))
    return sum(scores) / len(gold)

Common pitfalls

• Class imbalance is significant in legal documents, making unweighted F1 misleading.
• Sentence boundary detection errors in raw text can misalign predictions with gold labels.

Evidence (verbatim from paper)

The rhetorical roles task (a multiclass prediction problem) is evaluated using a weighted F1 score based on the test data.

Citation

@misc{modi2023legaleval,
  title={SemEval 2023 Task 6: LegalEval - Understanding Legal Texts},
  author={Modi et al. (2023)},
  year={2023},
  note={arXiv:2304.09548}
}

• arXiv: 2304.09548