Downstream Eval

downstream-eval

Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity — Yeongbin Seo et al. (2025) (arXiv:2509.18577, 2025)

What this evaluates

Evaluates the downstream language and symbolic capabilities of LLMs pretrained on filtered web corpora. It probes general knowledge, reasoning, comprehension, and code/math problem-solving to assess how different data filtering strategies impact model performance.

Datasets

• Dolma (v1.6) — total 6300000000; splits: train (3000000000), test (-1)
• Pile-github — total 6000000000; splits: train (3000000000), test (-1); repo https://www.kaggle.com/datasets/dschettler8845/the-pile-github-files-part-01

Metrics

• average normalized accuracy (primary) — range: [0, 1] or percent
  • Normalized per-task accuracy is computed as $a_{n}=(a_{m}-a_{r})/(1-a_{r})$, where $a_{m}$ is model accuracy and $a_{r}$ is random guessing accuracy. The final score is the arithmetic mean of $a_{n}$ across all evaluated benchmarks.

Input / output format

Input: Filtered pretraining corpus blocks (512 tokens) for model training; downstream benchmark prompts for evaluation.

Output: Model-generated predictions or answers for each downstream benchmark task.

Scoring recipe

def normalized_accuracy(model_acc, random_acc):
    return (model_acc - random_acc) / (1 - random_acc)

def compute_metric(predictions, gold, random_acc):
    task_acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    return normalized_accuracy(task_acc, random_acc)

# Final score: mean of task normalized accuracies across all benchmarks

Common pitfalls

• Excludes downstream benchmarks where baseline normalized accuracy < 0.001 for 1.5B models, reducing the total count from the full gauntlet.
• DSIR baseline is excluded for Pile-github experiments due to the difficulty of selecting an appropriate reference dataset.
• Normalization requires knowing the exact random guessing accuracy ($a_r$) per task, which varies by benchmark format.

Evidence (verbatim from paper)

We normalize the accuracy of the individual task as $a_{n}=rac{a_{m}-a_{r}}{1-a_{r}}$, where $a_{m}$ is the accuracy of the model and $a_{r}$ is the expected accuracy of random guessing. We report the average normalized accuracy for each task, task category, and the average across all categories.

Citation

@misc{seo2025priorbasednoisy,
  title={Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity},
  author={Yeongbin Seo et al. (2025)},
  year={2025},
  note={arXiv:2509.18577}
}

• arXiv: 2509.18577