Capbencher Eval

capbencher-eval

How Can I Publish My LLM Benchmark Without Giving the True Answers Away? — Ishida et al. (2025) (arXiv:2505.18102, 2025)

What this evaluates

Evaluates LLMs on standard benchmarks modified with randomized answers to measure performance tracking and detect data contamination via a Bayes accuracy ceiling. The protocol compares model accuracy against a predefined theoretical maximum (Bayes accuracy) to identify overfitting or memorization. It also assesses robustness to reverse-engineering attacks and cross-lingual contamination.

Datasets

• GSM8K — total ?; splits: test (-1)
• ARC-Challenge — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• MathQA — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• HLE-MC — total ?; splits: test (-1)
• MMLU-ProX — total ?; splits: test (-1)
• BoolQ — total ?; splits: test (-1)
• GPQA (diamond) — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• MATH-500 — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered questions compared to the ground truth. Evaluated on both capped (randomized) and non-capped benchmarks to track model progress.
• Kendall's tau — range: [-1, 1]
  • Rank correlation coefficient measuring how well the capped benchmark tracks model improvements relative to the non-capped benchmark.
• binomial test p-value — range: [0, 1]
  • Statistical test to determine if observed accuracy significantly exceeds the predefined Bayes accuracy ceiling, indicating data contamination.

Input / output format

Input: Question text, optionally with multiple-choice options. For capped benchmarks, includes a randomized answer and an instruction to randomly select a number from a list and add it to the solution.

Output: Direct answer string, selected choice label, or generated code snippet.

Scoring recipe

from scipy.stats import binom
def evaluate(predictions, golds, bayes_acc):
    n = len(golds)
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    acc = correct / n
    p_val = binom.sf(correct - 1, n, bayes_acc)
    return {'accuracy': acc, 'p_value': p_val, 'contaminated': p_val < 0.05}

Common pitfalls

• Bayes accuracy ceiling is task-specific and must be pre-defined by the benchmark creator (e.g., 50%, 25%, 10%).
• Contamination detection requires a statistical binomial test against the Bayes ceiling, not just checking if raw accuracy exceeds a fixed threshold.
• Reverse engineering prompts (P1/P2) can allow models to recover true answers, artificially inflating accuracy and masking contamination signals.

Evidence (verbatim from paper)

As expected, the accuracy of capped benchmarks increased as the accuracy of non-capped benchmarks increased (Kendall’s $ au$ is $0.96$), as shown in Fig.[3]. This demonstrates that CapBencher remain effective for tracking LLM progress. As shown in Fig.[4], our method detected contamination across all models and benchmarks, as indicated by accuracies surpassing the Bayes accuracy of 50%.

Citation

@misc{ishida2025capbencher,
  title={How Can I Publish My LLM Benchmark Without Giving the True Answers Away?},
  author={Ishida et al. (2025)},
  year={2025},
  note={arXiv:2505.18102}
}

• arXiv: 2505.18102