Mmlu Bbh Ifeval Eval

mmlu-bbh-ifeval-eval

Compute-Constrained Data Selection — Yin et al. (2024) (arXiv:2410.16208, 2024)

What this evaluates

Evaluates instruction-tuned language models on factual knowledge, complex reasoning, and instruction-following capabilities. The protocol measures how different data selection methods and model sizes impact performance under strict compute budgets.

Datasets

• MMLU — total ?; splits: test (-1)
• BBH — total ?; splits: test (-1)
• IFEval — total ?; splits: test (-1)

Metrics

• 5-shot accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered multiple-choice questions across 57 subjects using 5 in-context examples.
• 3-shot exact match score (primary) — range: [0, 1]
  • Percentage of tasks where the model's generated answer exactly matches the ground truth using 3 in-context examples.
• 0-shot accuracy (primary) — range: [0, 1]
  • Percentage of instruction-following prompts where the model correctly follows the specified constraints without in-context examples.

Input / output format

Input: Instruction-tuned finetuning data (curated from COT, Flan V2, Dolly, Open Assistant 1). Evaluation inputs are benchmark prompts formatted with 5-shot, 3-shot, or 0-shot examples depending on the task.

Output: Model-generated text responses to benchmark prompts, evaluated for exact match or multiple-choice correctness.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return correct / len(golds)

Common pitfalls

• Compute budget trade-offs between model size and data selection method are often misinterpreted as pure data quality effects.
• Pareto frontiers are fitted using power laws in linear-log space, which can obscure non-monotonic scaling behaviors.
• Few-shot evaluation settings (5-shot, 3-shot, 0-shot) must be strictly adhered to, as performance varies significantly with prompt length and in-context examples.

Evidence (verbatim from paper)

For MMLU, we report 5-shot accuracy; for BBH, we report 3-shot exact match score; and for IFEval, we report 0-shot accuracy.

Citation

@misc{yin2024computeconstrained,
  title={Compute-Constrained Data Selection},
  author={Yin et al. (2024)},
  year={2024},
  note={arXiv:2410.16208}
}

• arXiv: 2410.16208