Bbh Mmlu Predictability Eval

bbh-mmlu-predictability-eval

How predictable is language model benchmark performance? — Owen et al. (2024) (arXiv:2401.04757, 2024)

What this evaluates

This protocol evaluates how well aggregate and per-task benchmark performance can be predicted from scaled compute using scaling law fits. It probes the monotonicity and predictability of LLM capabilities across compute scaling, distinguishing between stable scaling trends and emergent or non-monotonic behaviors.

Datasets

• BIG-Bench Hard (BBH) — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• mean absolute error (primary) — range: percent
  • The absolute difference between the predicted benchmark accuracy (from a fitted sigmoid curve) and the actual reported accuracy, measured in percentage points. Averaged over held-out data points.

Input / output format

Input: Historical pairs of scaled compute (FLOP) and benchmark accuracy scores for a set of models.

Output: Predicted benchmark accuracy (%) for a target compute value, along with the absolute error relative to the true accuracy.

Scoring recipe

def compute_mean_absolute_error(compute_history, accuracy_history, held_out_compute, held_out_accuracy):
    model = SigmoidFit()
    model.fit(compute_history, accuracy_history)
    predictions = model.predict(held_out_compute)
    errors = [abs(pred - true_acc) for pred, true_acc in zip(predictions, held_out_accuracy)]
    return sum(errors) / len(errors)

Common pitfalls

• Confusing 'scaled compute' (FLOP × dataset size) with raw training FLOP.
• Assuming all tasks follow monotonic scaling; emergent capabilities cause sharp non-monotonic jumps that break sigmoid fits.
• Treating percentage point (pp) error as a probability or ratio rather than an absolute difference in accuracy.

Evidence (verbatim from paper)

predictions have average error of 3.9pp across a doubling of compute (0.33 orders of magnitude), rising to 12.5pp at two orders of magnitude scaling. ... Mean absolute error is 17pp, and the median absolute error is 8.4pp.

Citation

@misc{owen2024predictable,
  title={How predictable is language model benchmark performance?},
  author={Owen et al. (2024)},
  year={2024},
  note={arXiv:2401.04757}
}

• arXiv: 2401.04757