Openexempt Eval

openexempt-eval

OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand — Servantez et al. (2026) (arXiv:2601.13183, 2026)

What this evaluates

Probes legal reasoning capabilities in U.S. bankruptcy exemption law, specifically testing multi-step inference, robustness to distractors and obfuscation, and scalability across asset counts and temporal complexity.

Datasets

• OpenExempt — total 9765; splits: dev (465), test (9300); repo https://github.com/servantez/OpenExempt

Metrics

• macro-averaged F1 (primary) — range: [0, 1]
  • Computed at the sample level, then macro-averaged across samples. For asset-level tasks, per-asset scores are averaged to determine the sample score. For multi-label tasks, set overlap is used. Unparseable outputs are scored as incorrect.
• MARE — range: [0, 1]
  • Mean absolute relative error for dollar-valued predictions. A prediction is correct if it falls within a 5% tolerance: I_tau(y_hat, y) = 1[|y_hat/(y+epsilon) - 1| < tau] with epsilon=1 and tau=0.05.

Input / output format

Input: Natural language legal problem descriptions based on U.S. Bankruptcy Code statutes, often containing obfuscating or distractor statements, with varying complexity parameters (e.g., number of domiciles, asset count).

Output: Structured predictions (e.g., exemption citations, asset valuations, dollar amounts) or natural language answers. Outputs must be parseable; unparseable responses are marked invalid and scored as incorrect.

Scoring recipe

def score(predictions, golds):
    sample_scores = []
    for pred, gold in zip(predictions, golds):
        if not is_valid_format(pred):
            sample_scores.append(0.0)
            continue
        if is_multilabel(gold):
            score = set_overlap(pred, gold)
        elif is_dollar_valued(gold):
            score = 1 if abs(pred/(gold+1) - 1) < 0.05 else 0
        else:
            score = compute_f1(pred, gold)
        sample_scores.append(score)
    return macro_average(sample_scores)

Common pitfalls

• Ignoring format compliance: unparseable structured outputs are automatically scored as incorrect, which can significantly impact F1.
• Asset-level aggregation: For tasks like EC/EV, failing to average per-asset scores before computing the sample score will cause assets with more exemptions to dominate the metric.
• MARE instability near zero: Not using the stabilizing constant (epsilon=1) when computing relative error for small gold amounts leads to unstable or unbounded error values.

Evidence (verbatim from paper)

Across all tasks, OpenExempt reports precision, recall and F1 scores computed at the sample level and then macro-averaged across samples. For asset-level tasks (EC, EV), the evaluator first computes per asset scores within a case, then averages across assets to determine the sample score, preventing assets with more applicable exemptions from dominating the aggregate. For tasks with multi-label predictions (AE, EC, EV), we evaluate using set overlap across discrete labels (jurisdictions or exemption citations). For tasks involving dollar valued predictions (EV, NA, OE), we additionally compute mean absolute relative error (MARE) between predicted and gold amounts. A numeric prediction is treated as correct if it falls within a 5% absolute relative error tolerance of the corresponding gold value.

Citation

@misc{servantez2026openexempt,
  title={OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand},
  author={Servantez et al. (2026)},
  year={2026},
  note={arXiv:2601.13183}
}

• arXiv: 2601.13183