Sci Verifybench Eval

sci-verifybench-eval

SCI-Verifier: Scientific Verifier with Thinking — Zheng et al. (2025) (arXiv:2509.24285, 2025)

What this evaluates

Evaluates a model's ability to perform cross-disciplinary scientific verification by judging the correctness or equivalence of proposed answers to scientific problems. It probes domain-specific logical reasoning, handling of complex mathematical/scientific transformations, and robustness to prompt variations.

Datasets

• SCI-VerifyBench — total ?; splits: test (-1); repo https://github.com/Zhengsh123/SCI-Verifier

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly verified samples out of the total. Positive and negative samples are balanced by construction.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for the verification task, reported alongside Accuracy on VerifierBench and VerifyBench-Hard.

Input / output format

Input: A scientific problem or statement paired with a proposed answer, often containing domain-specific equivalence transformations (e.g., formula rewrites, logical substitutions).

Output: A binary verification judgment (correct/incorrect) accompanied by chain-of-thought reasoning.

Scoring recipe

accuracy = sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(golds)
f1 = f1_score(golds, predictions, average='binary')
return accuracy, f1

Common pitfalls

• Prompt sensitivity: general models lack an intrinsic notion of answer equivalence and rely heavily on prompt phrasing, causing performance drops when prompts are modified.
• Domain-dependent difficulty: Math and Physics tasks are significantly harder due to complex transformations (e.g., factorization, Taylor expansions), leading to lower scores compared to Biology/Chemistry/QA.
• Model scale is not decisive: Larger models do not consistently outperform smaller ones on verification tasks because the task relies more on prior knowledge and targeted reasoning than raw capacity.

Evidence (verbatim from paper)

For evaluation, we report Accuracy on SCI-VerifyBench, since positive and negative samples are balanced by construction. On VerifierBench and VerifyBench-hard, we additionally report F1 score alongside Accuracy. In all cases, higher values indicate stronger verification performance.

Citation

@misc{zheng2025sciverifier,
  title={SCI-Verifier: Scientific Verifier with Thinking},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2509.24285}
}

• arXiv: 2509.24285