Dclm Benchmark Eval

dclm-benchmark-eval

DataComp-LM: In search of the next generation of training sets for language models — Li et al. (2024) (arXiv:2406.11794, 2024)

What this evaluates

Evaluates the effectiveness of data curation strategies for language models by training base models on curated corpora and measuring performance on 53 downstream tasks. It isolates data quality effects from architectural and computational variables using fixed training recipes across multiple compute scales.

Datasets

• DCLM downstream tasks — total ?; splits: test (-1); repo https://github.com/mlfoundations/dclm

Metrics

• MMLU 5-shot accuracy (primary) — range: [0, 1]
  • Standard 5-shot multiple-choice accuracy on the MMLU benchmark. Computed as the fraction of correctly predicted answers given five in-context examples.
• CORE centered accuracy — range: [0, 1]
  • Centered accuracy computed over a subset of 22 tasks. Accuracy per task is linearly rescaled so that 0 corresponds to random guessing and 1 corresponds to perfect accuracy, then averaged.
• EXTENDED centered accuracy — range: [0, 1]
  • Averages the centered accuracy across all 53 evaluation tasks in the suite.

Input / output format

Input: Base decoder-only Transformer model (trained on curated data) evaluated on 53 downstream NLP tasks without fine-tuning. Inputs vary by task (e.g., multiple-choice questions, open-ended prompts).

Output: Model-generated predictions or token probabilities for each downstream task instance, aggregated into per-task accuracy scores.

Scoring recipe

def compute_task_accuracy(preds, golds):
    return sum(p == g for p, g in zip(preds, golds)) / len(golds)

def centered_acc(task_acc, random_acc):
    return (task_acc - random_acc) / (1.0 - random_acc)

# CORE: 22 tasks
core_scores = [centered_acc(compute_task_accuracy(p, g), random_acc) for p, g in CORE_TASKS]
core_acc = sum(core_scores) / len(core_scores)

# EXTENDED: 53 tasks
ext_scores = [centered_acc(compute_task_accuracy(p, g), random_acc) for p, g in ALL_TASKS]
ext_acc = sum(ext_scores) / len(ext_scores)

Common pitfalls

• Test set contamination: Training data often overlaps with evaluation sets; the benchmark requires explicit decontamination reports and may re-evaluate top submissions for overlap.
• Compute scale sensitivity: Data curation rankings can shift across different model sizes and Chinchilla multipliers (400M to 7B), though high rank correlation is observed.
• Base model restriction: Evaluation must be performed on base models without fine-tuning, unlike many standard LLM benchmarks.

Evidence (verbatim from paper)

To evaluate data curation algorithms, we focus on three main performance metrics. First, we consider MMLU 5-shot accuracy [78], which is widely used to compare state-of-the-art models like GPT-4 [122] and Llama 3 70B [4]. Second, we propose the CORE centered accuracy, computed over a subset of 22 tasks (e.g., HellaSwag [195] and ARC-E [43]) that provide a low-variance signal even at small scales, linearly rescaling the accuracy per task so that 0 corresponds to random guessing and 1 corresponds to perfect accuracy. Finally, we report the EXTENDED centered accuracy, which averages the centered performance for all of our 53 tasks.

Citation

@misc{li2024datacomplm,
  title={DataComp-LM: In search of the next generation of training sets for language models},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2406.11794}
}

• arXiv: 2406.11794