Helm Lite Eval

helm-lite-eval

Pitfalls of Evaluating Language Models with Open Benchmarks — Hasan et al. (2025) (arXiv:2507.00460, 2025)

What this evaluates

This evaluation probes the robustness of open benchmarks against test-set memorization and data leakage. It measures whether small language models can artificially inflate leaderboard scores by overfitting directly to public test sets, revealing flaws in current benchmarking practices.

Datasets

• HELM-lite — total ?; splits: test (-1)

Metrics

• Exact Match (primary) — range: [0, 1]
  • 1 if the predicted answer exactly matches the gold answer (after normalization), else 0. Averaged over the test set.
• Quasi Exact Match — range: [0, 1]
  • Matches the gold answer after normalizing whitespace and case, as defined in the HELM benchmark.
• F1 Score — range: [0, 1]
  • Token-level harmonic mean of precision and recall for open-ended QA tasks.
• BLEU-4 — range: [0, 100]
  • 4-gram BLEU score for machine translation evaluation.
• Equivalent (CoT) — range: [0, 1]
  • Checks if the predicted answer is mathematically equivalent to the gold answer, often leveraging chain-of-thought reasoning.

Input / output format

Input: Text prompts from HELM-lite scenarios, including multiple-choice questions, open-ended QA, translation pairs, and math problems.

Output: Model-generated answers or translations, formatted according to each scenario's specific requirements.

Scoring recipe

def score(predictions, golds, metric):
    if metric in ["Exact Match", "Quasi Exact Match"]:
        return sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g)) / len(golds)
    elif metric == "F1 Score":
        return token_f1(predictions, golds)
    elif metric == "BLEU-4":
        return bleu_4(predictions, golds)
    elif metric == "Equivalent (CoT)":
        return math_equiv(predictions, golds)
    return 0.0

Common pitfalls

• Training models on public test sets (data leakage) artificially inflates scores without reflecting true generalization.
• Different scenarios use different evaluation metrics, making direct cross-benchmark comparison difficult.

Evidence (verbatim from paper)

Each scenario is evaluated using metrics defined in the original HELM benchmark. MMLUExact MatchClaude 3.5 Sonnet80.9088.5091.6096.2094.8094.24

Citation

@misc{hasan2025pitfallsevaluating,
  title={Pitfalls of Evaluating Language Models with Open Benchmarks},
  author={Hasan et al. (2025)},
  year={2025},
  note={arXiv:2507.00460}
}

• arXiv: 2507.00460