# Mosaicml Eval

> Evaluates downstream language model capabilities across 33 question-answering tasks. It measures how effectively data pruning strategies improve general performance compared to unpruned baselines, using a normalized accuracy metric that accounts for random guessing baselines. Use when the user wants to benchmark on MosaicML evaluation gauntlet, or asks about evaluating this task. Reports average normalized accuracy.

- Skill: `qhjqhj00/mosaicml-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mosaicml-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mosaicml-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mosaicml-eval

---


# mosaicml-eval

> Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models — Ankner et al. (2024) (arXiv:2405.20541, 2024)

## What this evaluates

Evaluates downstream language model capabilities across 33 question-answering tasks. It measures how effectively data pruning strategies improve general performance compared to unpruned baselines, using a normalized accuracy metric that accounts for random guessing baselines.

## Datasets

- **MosaicML evaluation gauntlet** — total 33; splits: test (-1)

## Metrics

- `average normalized accuracy` **(primary)** — range: percent
  - For each task, normalize accuracy as $a_n = (a_m - a_r) / (1 - a_r)$, where $a_m$ is model accuracy and $a_r$ is random guessing accuracy. Then average $a_n$ across all tasks or task categories.

## Input / output format

**Input**: Question-answering prompts and options from the 33 tasks in the MosaicML evaluation gauntlet.

**Output**: Model-generated answer or selected option for each question.

## Scoring recipe

```python
def compute_avg_normalized_accuracy(task_accuracies, random_guess_accuracies):
    normalized_scores = []
    for acc, r_acc in zip(task_accuracies, random_guess_accuracies):
        norm_acc = (acc - r_acc) / (1 - r_acc)
        normalized_scores.append(norm_acc)
    return sum(normalized_scores) / len(normalized_scores)
```

## Common pitfalls

- Using test-set perplexity on the original pretraining data as a proxy for downstream performance is misleading; pruned data often worsens test perplexity while improving downstream accuracy.
- Averaging raw accuracies across tasks without normalizing by each task's random guessing baseline will bias results toward tasks with higher chance accuracy.

## Evidence (verbatim from paper)

> We evaluate models on 33 different downstream question-answering tasks using the MosaicML evaluation gauntlet (MosaicML, 2023a). Before averaging the accuracy across tasks, we normalize each task by the baseline of random guessing . Specifically,we normalize the accuracy of each individual task as $a_n = rac{a_m - a_r}{1 - a_r}$ ,where $a_m$ is the accuracy of the model and $a_r$ is the expected accuracy of random guessing. We report the average normalized accuracy for each task category as well as the average normalized accuracy across all task categories.

## Citation

```bibtex
@misc{ankner2024perplexed,
  title={Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models},
  author={Ankner et al. (2024)},
  year={2024},
  note={arXiv:2405.20541}
}
```

- arXiv: 2405.20541