Mga Pretraining Eval

mga-pretraining-eval

Reformulation for Pretraining Data Augmentation — Hao et al. (2025) (arXiv:2502.04235, 2025)

What this evaluates

This evaluation protocol assesses the effectiveness of a genre-audience reformulation data augmentation strategy on large language model pretraining. It measures downstream task performance across a suite of reasoning, knowledge, and language understanding benchmarks to determine if augmented data improves generalization and mitigates data repetition degradation.

Datasets

• ARC, HellaSwag, Winogrande, MMLU, GSM8K, CSQA, OpenBookQA, PIQA, TriviaQA — total ?; splits: test (-1)

Metrics

• average benchmark accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy calculated over multiple-choice and generation tasks. Evaluated in 0-shot format for all benchmarks except MMLU, which uses 5-shot. The primary reported metric is the average accuracy across 12 benchmarks.

Input / output format

Input: Multiple-choice questions or open-ended prompts formatted for 0-shot or 5-shot evaluation, provided as text prompts to the language model.

Output: Model-generated text or selected option corresponding to the prompt.

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return correct / len(gold_labels)

# Note: MMLU uses 5-shot, others use 0-shot. Final score is average across 12 benchmarks.

Common pitfalls

• Relying solely on validation loss to judge model quality, as the paper explicitly demonstrates that higher validation loss does not correlate with degraded benchmark performance.
• Ignoring shot settings: MMLU requires 5-shot evaluation while all other benchmarks use 0-shot, which significantly impacts accuracy scores.
• Comparing models trained on different data recipes or token counts without normalizing for the total training budget (e.g., 600B vs 1T tokens).

Evidence (verbatim from paper)

We follow popular practice of LightEval [[24]] and LM-Harness [[25]], evaluate on a comprehensive suite of open benchmarks include ARC-Easy/Challenge*[[26]], HellaSwag[[27]], Winogrande[[28]], MMLU[[29]], GSM8K[[30]]*, etc. For training dynamics, we report the average of 12 benchmarks and validation losses on held-out fineweb-edu-dedup data. All benchmarks are 0-shot evaluations (obtained through LightEval), except for MMLU (5-shot).

Citation

@misc{hao2025reformulation,
  title={Reformulation for Pretraining Data Augmentation},
  author={Hao et al. (2025)},
  year={2025},
  note={arXiv:2502.04235}
}

• arXiv: 2502.04235