Lexrel Eval

lexrel-eval

LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases — Cai et al. (2025) (arXiv:2512.12643, 2025)

What this evaluates

Probes large language models' ability to extract structured legal relations (relation types and factual arguments) from Chinese civil court judgments. It evaluates both zero-shot prompting and fine-tuning capabilities, while also measuring performance on long-tail relation types and downstream legal reasoning tasks.

Datasets

• LexRel — total 1140; splits: test (-1)

Metrics

• precision — range: [0, 1]
  • Ratio of correctly predicted legal relation types or arguments to the total number of predicted instances.
• recall — range: [0, 1]
  • Ratio of correctly predicted legal relation types or arguments to the total number of gold-standard instances.
• micro-F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed globally across all instances in the dataset.
• macro-F1 — range: [0, 1]
  • Harmonic mean of precision and recall computed by averaging the F1 scores across all 265 relation types.

Input / output format

Input: Chinese civil court judgment texts (either factual text or full judgment text depending on the baseline setting).

Output: Extracted legal relation types and corresponding argument spans/text for each relation.

Scoring recipe

def score_type_extraction(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g)
    fp = len(predictions) - tp
    fn = len(gold) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

def score_argument_extraction(predictions, gold):
    matches = [1 if judge_llm(p, g) else 0 for p, g in zip(predictions, gold)]
    tp = sum(matches)
    fp = len(predictions) - tp
    fn = len(gold) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

Common pitfalls

• Argument extraction uses an LLM-as-a-Judge for semantic equivalence rather than exact string matching, which requires careful prompt engineering to replicate.
• Macro-F1 is calculated by averaging F1 across all 265 relation types, heavily penalizing models on long-tail categories that micro-F1 masks.
• Downstream task scores (0-100 scale) follow LawBench guidelines for correctness and completeness, not standard classification metrics.

Evidence (verbatim from paper)

We use precision, recall, micro-F1 score and macro-F1 score (computing by averaging F1 scores across different relation types) to assess model performance on both type extraction and argument extraction tasks. For type extraction, evaluation is conducted by directly matching each predicted legal relation type against the gold-standard label. Considering the inherent ambiguity in legal language and the cost of large-scale human evaluation, we adopt a LLM-as-a-Judge strategy for evaluating argument extraction correctness.

Citation

@misc{cai2025lexrel,
  title={LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2512.12643}
}

• arXiv: 2512.12643