Verisoftbench Eval

verisoftbench-eval

VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean — Xin et al. (2026) (arXiv:2602.18307, 2026)

What this evaluates

This benchmark probes an AI system's ability to perform repository-scale formal verification in Lean 4. It specifically tests context-aware proof automation, measuring how well models handle project-specific abstractions and transitive dependency closures beyond standard mathematical libraries.

Datasets

• VeriSoftBench-Full — total 500; splits: test (500); repo https://github.com/utopia-group/VeriSoftBench

Metrics

• solve_rate (primary) — range: [0, 1]
  • Percentage of tasks marked as “Solved”. A task is solved if at least one of the k sampled candidate proof scripts, or any of their subsequent repair iterations, successfully passes Lean 4 verification.

Input / output format

Input: Per instance: the target proof obligation/goal, the repository context (files, lemmas, dependencies), and for repair rounds: compiler error messages, relevant code lines, and the current proof state.

Output: A tactical proof block/script intended to satisfy the target goal.

Scoring recipe

solved_count = 0
for task in dataset:
    success = False
    for _ in range(k):
        script = model.generate(task.context, task.goal)
        if verify_lean4(script, task.repo_toolchain, timeout=300):
            success = True; break
        for _ in range(r):
            err = get_compiler_feedback(script)
            script = model.repair(script, err, task.context)
            if verify_lean4(script, task.repo_toolchain, timeout=300):
                success = True; break
        if success: break
    if success: solved_count += 1
return solved_count / len(dataset)

Common pitfalls

• Must use the exact Lean 4 toolchain version associated with each source repository; pre-building environments and caching large dependencies like Mathlib is required for stable evaluation.
• Context presentation differs between VeriSoftBench-Full and VeriSoftBench-Aristotle; the latter requires a pre-compiled environment and includes same-file lemmas, which significantly impacts prover performance.
• Verification timeout is strictly 300 seconds per candidate script; exceeding this counts as a failure.

Evidence (verbatim from paper)

A task is marked as “Solved” if any of the $k$ candidates or their subsequent repairs pass verification. For each evaluation run, the model receives a task associated with the selected contextual configuration as input, and the evaluation proceeds in three stages: (1) we sample $k$ candidate proof scripts from the model. Each candidate is generated as a tactical proof block intended to satisfy the target goal; (2) we wrap each generated script into its original file context and attempt verification using the Lean 4 compiler. To ensure a stable and efficient evaluation, the all task environments are pre-built prior to verification; we use the specific toolchain version associated with the source repository of each task and enforce a 300-second compilation timeout per candidate

Citation

@misc{xin2026verisoftbench,
  title={VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean},
  author={Xin et al. (2026)},
  year={2026},
  note={arXiv:2602.18307}
}

• arXiv: 2602.18307