Ultra Fineweb Eval

ultra-fineweb-eval

Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data — Wang et al. (2025) (arXiv:2505.05427, 2025)

What this evaluates

Evaluates the quality of pre-training datasets by measuring the downstream performance of models trained on them. It probes general language understanding, commonsense reasoning, and multilingual capabilities through standard zero-shot benchmarks.

Datasets

• MMLU — total ?; splits: test (-1)
• ARC-C — total ?; splits: test (-1)
• ARC-E — total ?; splits: test (-1)
• CommonSenseQA — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• OpenbookQA — total ?; splits: test (-1)
• PIQA — total ?; splits: test (-1)
• SIQA — total ?; splits: test (-1)
• Winogrande — total ?; splits: test (-1)
• C-Eval — total ?; splits: test (-1)
• CMMLU — total ?; splits: test (-1)

Metrics

• AverageEnglish — range: percent
  • Arithmetic mean of zero-shot accuracy scores across the nine English benchmarks (MMLU, ARC-C, ARC-E, CommonSenseQA, HellaSwag, OpenbookQA, PIQA, SIQA, Winogrande).
• AverageChinese — range: percent
  • Arithmetic mean of zero-shot accuracy scores across the two Chinese benchmarks (C-Eval, CMMLU).
• Average (primary) — range: percent
  • Arithmetic mean of zero-shot accuracy scores across all listed English and Chinese benchmarks.

Input / output format

Input: Zero-shot multiple-choice prompts from standard benchmarks fed to a trained LLM.

Output: Model-generated token predictions used to compute exact-match accuracy against provided answer choices.

Scoring recipe

all_scores = []
for benchmark in [MMLU, ARC-C, ARC-E, CommonSenseQA, HellaSwag, OpenbookQA, PIQA, SIQA, Winogrande, C-Eval, CMMLU]:
    acc = compute_zero_shot_accuracy(model, benchmark.test_set)
    all_scores.append(acc)
average = sum(all_scores) / len(all_scores)
return average

Common pitfalls

• The evaluation strictly uses a zero-shot setting via the Lighteval library; applying few-shot prompting or instruction tuning during evaluation will invalidate the reported scores.
• Scores are reported as percentage points (pp) averaged across tasks; misinterpreting them as raw accuracy, median scores, or task-specific scores will lead to incorrect comparisons.
• The 'Average' metric combines English and Chinese benchmarks; separating them requires using AverageEnglish and AverageChinese instead.

Evidence (verbatim from paper)

The evaluation metrics include:

• •

  AverageEnglish: Average score across standard English metrics including MMLU*(Hendrycks et al., [2020]), ARC-C(Clark et al., [2018]), ARC-E(Clark et al., [2018]), CommonSenseQA(Talmor et al., [2018]), HellaSwag(Zellers et al., [2019]), OpenbookQA(Mihaylov et al., [2018]), PIQA(Bisk et al., [2020]), SIQA(Sap et al., [2019]), and Winogrande(Sakaguchi et al., [2021])*.

• •

  AverageChinese: Average score of Chinese metrics, including C-Eval*(Huang et al., [2023])* and CMMLU*(Li et al., [2023])*.

• •

  Average: The combined average score of all the above evaluation metrics.

Citation

@misc{wang2025ultrafineweb,
  title={Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.05427}
}

• arXiv: 2505.05427