qhjqhj00 nllp-2024-legal-nli-eval
Augmenting Legal Decision Support Systems with LLM-based NLI for Analyzing Social Media Evidence — Kadiyala et al. (2024) (arXiv:2410.15990, 2024)
What this evaluates
Probes an LLM's ability to perform Natural Language Inference (NLI) within the legal domain. Specifically, it tests whether the model can correctly classify the logical relationship (entailment, neutral, or contradiction) between a formal legal case summary and an informal social media review.
Datasets
- NLLP 2024 Legal NLI — total 396; splits: train (312), test (84); repo https://github.com/1-800-SHARED-TASKS/EMNLP-2024-NLLP
Metrics
F1(primary) — range: [0, 1]- Macro-averaged F1 score across the three NLI classes (entailment, neutral, contradiction). Computed as the harmonic mean of precision and recall for each class, then averaged.
Input / output format
Input: A legal premise (summary of a resolved class-action case) paired with a hypothesis (an online media text or social media review).
Output: A single class label: 'entailment', 'neutral', or 'contradiction'.
Scoring recipe
def compute_f1(predictions, gold):
classes = ['entailment', 'neutral', 'contradiction']
f1s = []
for c in classes:
tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
f1s.append(f1)
return sum(f1s) / len(f1s)
Common pitfalls
- The test set is very small (84 samples), making reported F1 scores highly sensitive to random initialization and data shuffling.
- Social media hypotheses contain informal language and semantic ambiguity, which disproportionately hurts contradiction detection compared to standard NLI benchmarks.
- Class distribution in the test set is skewed toward entailment (47.6%) and neutral (34.5%), so accuracy alone can be misleading; macro-F1 is required to evaluate minority class performance.
Evidence (verbatim from paper)
The paper presents a winning entry in the NLLP 2024 Legal NLI shared task, achieving state-of-the-art performance by fine-tuning large language models (LLMs) on legal premise-hypothesis pairs from class-action cases and social media reviews. A key innovation is the use of multi-stage training and ORPO-based alignment strategies, with GEMMA-2-27B and Mistral-8x7B outperforming others via preferred alignment (ORPO Preferred), achieving up to 0.887 F1.
Citation
@misc{kadiyala2024augmenting,
title={Augmenting Legal Decision Support Systems with LLM-based NLI for Analyzing Social Media Evidence},
author={Kadiyala et al. (2024)},
year={2024},
note={arXiv:2410.15990}
}
- arXiv: 2410.15990