Big Bench Predictability Eval

big-bench-predictability-eval

How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench — Ye et al. (2023) (arXiv:2305.14947, 2023)

What this evaluates

Evaluates the predictability of LLM performance across diverse BIG-bench tasks using an MLP-based predictor. It probes how well model scale, task type, and in-context examples correlate with actual benchmark scores, and tests robustness under different holdout strategies.

Datasets

• BIG-bench — total ?; splits: test (-1); repo https://github.com/google/BIG-bench

Metrics

• R² (primary) — range: [0, 1]
  • Coefficient of determination measuring the proportion of variance in actual BIG-bench scores explained by the MLP predictor. Calculated as 1 - (SS_res / SS_tot).
• RMSE — range: [0, 1]
  • Root Mean Squared Error between predicted and actual task performance scores.

Input / output format

Input: Task identifier, model scale/architecture parameters, and in-context example configurations.

Output: Predicted performance score (e.g., accuracy or normalized score) for each BIG-bench task.

Scoring recipe

def compute_metrics(predictions, actuals):
    mean_actual = sum(actuals) / len(actuals)
    ss_res = sum((p - a)**2 for p, a in zip(predictions, actuals))
    ss_tot = sum((a - mean_actual)**2 for a in actuals)
    r2 = 1 - (ss_res / ss_tot)
    rmse = (ss_res / len(actuals))**0.5
    return r2, rmse

Common pitfalls

• Predictability drops significantly under Cartesian product holdout, indicating high sensitivity to distribution assumptions.
• Emergent tasks are harder to predict but remain partially predictable when related tasks are included in the training set.

Evidence (verbatim from paper)

Large language model (LLM) performance on BIG-bench is highly predictable, with an MLP-based predictor achieving >95% R² and <0.05 RMSE, revealing learnable patterns across model scale, task, and in-context examples.

Citation

@misc{ye2023predictable,
  title={How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2305.14947}
}

• arXiv: 2305.14947