Meta Rater Eval

meta-rater-eval

Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models — Zhuang et al. (2025) (arXiv:2504.14194, 2025)

What this evaluates

Evaluates the downstream performance of language models pre-trained on data selected by various quality-based methods compared to random sampling. It probes how different data curation strategies impact general knowledge, commonsense reasoning, and reading comprehension capabilities.

Datasets

• ARC-Challenge, ARC-Easy, SciQ, HellaSwag, SIQA, WinoGrande, RACE, OpenbookQA — total ?; splits: test (-1)

Metrics

• average accuracy (primary) — range: percent
  • Arithmetic mean of per-task accuracy scores across eight downstream benchmarks. Computed as the sum of individual task accuracies divided by the number of tasks.

Input / output format

Input: Downstream task prompts formatted for in-context learning via lm-evaluation-harness.

Output: Model-generated predictions (multiple-choice answers or text completions) for each prompt.

Scoring recipe

accuracies = []
for task in [ARC_Challenge, ARC_Easy, SciQ, HellaSwag, SIQA, WinoGrande, RACE, OpenbookQA]:
    preds = model.generate(task.prompts, in_context=True)
    acc = compute_accuracy(preds, task.golds)
    accuracies.append(acc)
return sum(accuracies) / len(accuracies)

Common pitfalls

• The evaluation uses in-context learning via lm-evaluation-harness rather than zero-shot or fine-tuning, which may inflate or deflate scores differently than other protocols.
• The 'Average' metric is a simple arithmetic mean across tasks of varying difficulty and scale, which can mask significant performance drops in specific categories like Reading Comprehension.
• Data selection methods sample fixed token counts (e.g., 30B) while maintaining domain proportions, meaning the actual number of documents and domain mix varies across methods.

Evidence (verbatim from paper)

To comprehensively assess the capabilities of pre-trained models, we conduct holistic evaluations on various downstream tasks covering three significant categories: General Knowledge (including ARC-Challenge Clark et al. ([2018]), ARC-Easy, and SciQ Welbl et al. ([2017])), Commonsense Reasoning (including HellaSwag Zellers et al. ([2019]), SIQA Sap et al. ([2019]), and WinoGrande Sakaguchi et al. ([2020])), and Reading Comprehension (including RACE Lai et al. ([2017]) and OpenbookQA Mihaylov et al. ([2018])). Evaluations are conducted using the lm-evaluation-harness Gao et al. ([2023])) framework with in-context learning setting, and average accuracy is reported for convenient comparison.

Citation

@misc{zhuang2025metarater,
  title={Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models},
  author={Zhuang et al. (2025)},
  year={2025},
  note={arXiv:2504.14194}
}

• arXiv: 2504.14194