Tinybenchmarks Sampling Eval

tinybenchmarks-sampling-eval

tinyBenchmarks: evaluating LLMs with fewer examples — Felipe Maia Polo et al. (arXiv:2402.14992, 2024)

What this evaluates

Evaluates the efficiency and accuracy of LLM benchmarking by testing how well a small, strategically selected subset of examples predicts overall model performance on standard evaluation scenarios.

Datasets

• HELM — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• AlpacaEval 2.0 — total ?; splits: test (-1)
• Open LLM Leaderboard — total ?; splits: test (-1)

Metrics

• estimation error (primary) — range: percent
  • The absolute difference between the performance estimated from a curated subset of examples and the true performance on the full benchmark scenario. Calculated as |estimated_perf - true_perf|, typically reported as a percentage.

Input / output format

Input: A curated subset of evaluation examples $\widehat{\mathcal{I}}{j}$ from a benchmark scenario $\mathcal{I}{j}$, selected via stratified random sampling, K-Means clustering on correctness vectors, or IRT-based clustering.

Output: Estimated performance scores for evaluated LLMs on the full benchmark, derived using weighted aggregation of subset correctness or IRT latent ability models.

Scoring recipe

def scoring_recipe(predictions, gold):
    # predictions: estimated performance from subset
    # gold: true performance on full benchmark
    return abs(predictions - gold)

def estimate_performance(subset_correctness, weights):
    return sum(w * y for w, y in zip(weights, subset_correctness))

Common pitfalls

• Assuming uniform weights ($w_i = 1/|\widehat{\mathcal{I}}_j|$) are sufficient; cluster/IRT weights based on representative fractions yield better estimates.
• Ignoring distribution shifts in model correctness patterns over time, which can degrade clustering-based anchor point selection.
• Failing to account for the curse of dimensionality when using correctness vectors from a large training set for K-Means clustering.

Evidence (verbatim from paper)

showing that 100 carefully curated examples suffice to estimate performance on major benchmarks like MMLU, HELM, and Open LLM Leaderboard with <2% error.

Citation

@misc{polo2024tinybenchmarks,
  title={tinyBenchmarks: evaluating LLMs with fewer examples},
  author={Felipe Maia Polo et al.},
  year={2024},
  note={arXiv:2402.14992}
}

• arXiv: 2402.14992