Standard LLM Benchmarks Eval

standard-llm-benchmarks-eval

OLMoE: Open Mixture-of-Experts Language Models — Niklas Muennighoff et al. (2024) (arXiv:2409.02060, 2024)

What this evaluates

Evaluates language model capabilities across knowledge, reasoning, instruction following, and safety using a standard suite of downstream benchmarks. It measures both pretraining quality and post-adaptation performance on established NLP and coding tasks.

Datasets

• MMLU — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• ARC-Challenge — total ?; splits: test (-1)
• ARC-Easy — total ?; splits: test (-1)
• PIQA — total ?; splits: test (-1)
• WinoGrande — total ?; splits: test (-1)
• GSM8k — total ?; splits: test (-1)
• BBH — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)
• AlpacaEval 1.0 — total ?; splits: test (-1)
• XSTest — total ?; splits: test (-1)
• IFEval — total ?; splits: test (-1)

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • EM. Computed as the fraction of predictions that exactly match the gold answer after standard normalization.
• Pass@10 — range: [0, 1]
  • Fraction of generated code samples (out of 10 attempts) that pass all provided unit tests.
• win rate — range: percent
  • %win. Percentage of pairwise comparisons where the model's output is preferred over the reference by an LLM judge.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall computed over token-level matches between prediction and gold.
• loose accuracy — range: [0, 1]
  • Loose Acc. Fraction of predictions that match the gold answer according to a relaxed regex-based matching rule.

Input / output format

Input: Text prompts containing task instructions, optionally with few-shot examples (0, 3, 5, or 8-shot depending on the benchmark), or direct questions.

Output: Model-generated text completion or selected option string.

Scoring recipe

def score(predictions, golds, metric):
    if metric == 'EM':
        return sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g)) / len(golds)
    elif metric == 'Pass@10':
        return sum(1 for p in predictions[:10] if is_correct(p)) / 10
    elif metric == '%win':
        return judge_win_rate(predictions, reference_outputs)
    elif metric == 'F1':
        return compute_f1(predictions, golds)
    elif metric == 'Loose Acc':
        return sum(1 for p, g in zip(predictions, golds) if regex_match(p, g)) / len(golds)

Common pitfalls

• Varying few-shot settings across tasks (0-shot, 3-shot, 5-shot, 8-shot CoT) require strict prompt formatting to reproduce.
• AlpacaEval uses a reference-free pairwise comparison with an LLM judge, not fixed gold labels, making direct score replication sensitive to the judge model version.
• Loose accuracy for IFEval relies on specific regex patterns that may differ from strict exact-match implementations.

Evidence (verbatim from paper)

We run all evaluations ourselves with 5 few-shots, see [Appendix C] for details. Metric ($ ightarrow$) | EM | EM | EM | Pass@10 | %win | F1 | Loose Acc

Citation

@misc{muennighoff2024olmoe,
  title={OLMoE: Open Mixture-of-Experts Language Models},
  author={Niklas Muennighoff et al. (2024)},
  year={2024},
  note={arXiv:2409.02060}
}

• arXiv: 2409.02060