Optbench Eval

optbench-eval

OptProver: Bridging Olympiad and Optimization through Continual Training in Formal Theorem Proving — Li et al. (2026) (arXiv:2604.23712, 2026)

What this evaluates

Evaluates formal theorem proving capabilities specifically within the undergraduate optimization domain. It probes a model's ability to generate syntactically correct and semantically progressive Lean 4 proof steps or full scripts under strict verifier constraints, while measuring robustness against catastrophic forgetting on general math benchmarks.

Datasets

• OptBench — total 400; splits: test (400)
• MiniF2F-test — total ?; splits: test (-1)
• ProofNet-test — total ?; splits: test (-1)

Metrics

• Pass@1 — range: percent
  • Fraction of problems where the single generated proof attempt passes the Lean 4 verifier.
• Pass@32 (primary) — range: percent
  • Fraction of problems where at least one of the 32 generated proof attempts passes the Lean 4 verifier.
• Pass@256 — range: percent
  • Fraction of problems where at least one of the 256 generated proof attempts passes the Lean 4 verifier.

Input / output format

Input: Current Lean 4 proof state and goal statement.

Output: A Lean 4 tactic or complete proof script.

Scoring recipe

def pass_at_k(proof_attempts, verifier, k):
    solved = 0
    for attempts in proof_attempts:
        if any(verifier(attempt) for attempt in attempts[:k]):
            solved += 1
    return (solved / len(proof_attempts)) * 100

Common pitfalls

• Whole-proof and step-level provers use different evaluation budgets (Pass@32/256 vs Pass@1/32), making direct accuracy comparisons without context misleading.
• Syntactic correctness in Lean 4 does not guarantee proof progress; models often emit valid but stagnant tactics that fail to advance the goal.
• Pass@k measures the probability of finding at least one correct proof among k attempts, not the average number of correct proofs.

Evidence (verbatim from paper)

All whole-proof generation models are tested under Pass@32 and Pass@256. For step-level prover, we use an accumulative budget within Pass@1 and Pass@32.

Citation

@misc{li2026optprover,
  title={OptProver: Bridging Olympiad and Optimization through Continual Training in Formal Theorem Proving},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.23712}
}

• arXiv: 2604.23712