Benchmark Accuracy Eval

benchmark-accuracy-eval

Benchmarks Are Not That Out of Distribution: Word Overlap Predicts Performance — Chung et al. (2026) (arXiv:2602.10657, 2026)

What this evaluates

Evaluates zero-shot language model performance across a suite of 10 standard NLP benchmarks covering commonsense reasoning, science QA, and language modeling. It measures task accuracy and correlates it with word-level statistical overlap metrics to assess distributional alignment between pre-training data and evaluation sets.

Datasets

• ARC Easy — total ?; splits: test (-1)
• ARC Challenge — total ?; splits: test (-1)
• Hellaswag — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• SciQ — total ?; splits: test (-1)
• OpenBookQA — total ?; splits: test (-1)
• PIQA — total ?; splits: test (-1)
• lambada — total ?; splits: test (-1)
• SocialIQA — total ?; splits: test (-1)
• SWAG — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers out of the total number of evaluation instances.
• word-level unigram cross-entropy — range: other
  • Negative log-likelihood of benchmark unigrams under the pre-training corpus word frequency distribution. Computed as -sum(p_b(x) * log(p_p(x))) over the vocabulary.

Input / output format

Input: Zero-shot prompts for each benchmark task, formatted according to the original dataset specifications without in-context examples.

Output: Model's predicted answer or multiple-choice selection per instance.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_unigram_cross_entropy(benchmark_freq, pretrain_freq):
    entropy = 0.0
    for word in benchmark_freq:
        if word in pretrain_freq:
            entropy -= benchmark_freq[word] * math.log(pretrain_freq[word])
    return entropy

Common pitfalls

• Zero-shot evaluation may not reflect performance under fine-tuning or prompt engineering.
• Word-level cross-entropy measures distributional overlap, not semantic understanding, so high scores may stem from memorization rather than generalization.
• Cross-entropy is invariant to dataset size scaling, so it does not capture the effect of token exposure on learning signal strength.

Evidence (verbatim from paper)

We use 10 representative benchmarks, all evaluated in a zero-shot setting: ARC Easy, ARC Challenge, Hellaswag, MMLU, SciQ, OpenBookQA, PIQA, lambada, SocialIQA and SWAG. Examining 10 downstream benchmark performance alongside word-level unigram cross-entropy of benchmark data under pre-training corpus word frequency, we find a consistent negative correlation between benchmark performance and unigram cross-entropy.

Citation

@misc{chung2026benchmarks,
  title={Benchmarks Are Not That Out of Distribution: Word Overlap Predicts Performance},
  author={Chung et al. (2026)},
  year={2026},
  note={arXiv:2602.10657}
}

• arXiv: 2602.10657