# Big Bench Predictability Eval

> Evaluates the predictability of LLM performance across diverse BIG-bench tasks using an MLP-based predictor. It probes how well model scale, task type, and in-context examples correlate with actual benchmark scores, and tests robustness under different holdout strategies. Use when the user wants to benchmark on BIG-bench, or asks about evaluating this task. Reports R².

- Skill: `qhjqhj00/big-bench-predictability-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/big-bench-predictability-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/big-bench-predictability-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/big-bench-predictability-eval

---


# big-bench-predictability-eval

> How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench — Ye et al. (2023) (arXiv:2305.14947, 2023)

## What this evaluates

Evaluates the predictability of LLM performance across diverse BIG-bench tasks using an MLP-based predictor. It probes how well model scale, task type, and in-context examples correlate with actual benchmark scores, and tests robustness under different holdout strategies.

## Datasets

- **BIG-bench** — total ?; splits: test (-1); repo https://github.com/google/BIG-bench

## Metrics

- `R²` **(primary)** — range: [0, 1]
  - Coefficient of determination measuring the proportion of variance in actual BIG-bench scores explained by the MLP predictor. Calculated as 1 - (SS_res / SS_tot).
- `RMSE` — range: [0, 1]
  - Root Mean Squared Error between predicted and actual task performance scores.

## Input / output format

**Input**: Task identifier, model scale/architecture parameters, and in-context example configurations.

**Output**: Predicted performance score (e.g., accuracy or normalized score) for each BIG-bench task.

## Scoring recipe

```python
def compute_metrics(predictions, actuals):
    mean_actual = sum(actuals) / len(actuals)
    ss_res = sum((p - a)**2 for p, a in zip(predictions, actuals))
    ss_tot = sum((a - mean_actual)**2 for a in actuals)
    r2 = 1 - (ss_res / ss_tot)
    rmse = (ss_res / len(actuals))**0.5
    return r2, rmse
```

## Common pitfalls

- Predictability drops significantly under Cartesian product holdout, indicating high sensitivity to distribution assumptions.
- Emergent tasks are harder to predict but remain partially predictable when related tasks are included in the training set.

## Evidence (verbatim from paper)

> Large language model (LLM) performance on BIG-bench is highly predictable, with an MLP-based predictor achieving >95% R² and <0.05 RMSE, revealing learnable patterns across model scale, task, and in-context examples.

## Citation

```bibtex
@misc{ye2023predictable,
  title={How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2305.14947}
}
```

- arXiv: 2305.14947