qhjqhj00 legal-zero-days-eval
Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems — Greg Sadler et al. (2025) (arXiv:2508.10050, 2025)
What this evaluates
This benchmark probes an AI system's ability to detect previously undiscovered legal vulnerabilities within governance frameworks. It tests whether models can identify systemic flaws that could cause immediate disruption without requiring traditional litigation, measuring their capacity for advanced legal reasoning and regulatory logic parsing.
Datasets
- Legal Zero-Days — total ?; splits: test (-1)
Metrics
Accuracy(primary) — range: percent- Percentage of correctly identified legal vulnerabilities out of the total number of vulnerabilities or puzzles presented. Calculated as (correct identifications / total instances) × 100.
Input / output format
Input: A description of a legal puzzle or governance framework scenario containing an embedded, previously undiscovered legal vulnerability.
Output: A textual response identifying and describing the specific legal vulnerability within the provided scenario.
Scoring recipe
def calculate_accuracy(predictions, gold):
correct = 0
for pred, gold_vuln in zip(predictions, gold):
if is_correct_identification(pred, gold_vuln):
correct += 1
return (correct / len(gold)) * 100
Common pitfalls
- High performance variance across runs, indicated by large confidence intervals (e.g., ±13.50%), suggesting instability in vulnerability detection.
- Difficulty distinguishing between genuinely correct identifications and responses that merely mischaracterize or partially address the introduced flaws.
- Automated judge validation relied on a small ground-truth set (25 examples), which may not fully generalize to the broader benchmark.
Evidence (verbatim from paper)
Table 1 presents the performance of all evaluated models on our Legal Zero-Days benchmark. Gemini-2.5-pro-preview-05-06 achieved the highest accuracy at 10.00% ± 13.50%, followed by o3-2025-04-16 at 6.67% ± 9.70%. The remaining models performed considerably lower, with accuracy scores ranging from 1.85% to 5.19%.
Citation
@misc{sadler2025legalzerodays,
title={Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems},
author={Greg Sadler et al. (2025)},
year={2025},
note={arXiv:2508.10050}
}
- arXiv: 2508.10050