Mosaicml Eval

mosaicml-eval

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models — Ankner et al. (2024) (arXiv:2405.20541, 2024)

What this evaluates

Evaluates downstream language model capabilities across 33 question-answering tasks. It measures how effectively data pruning strategies improve general performance compared to unpruned baselines, using a normalized accuracy metric that accounts for random guessing baselines.

Datasets

• MosaicML evaluation gauntlet — total 33; splits: test (-1)

Metrics

• average normalized accuracy (primary) — range: percent
  • For each task, normalize accuracy as $a_n = (a_m - a_r) / (1 - a_r)$, where $a_m$ is model accuracy and $a_r$ is random guessing accuracy. Then average $a_n$ across all tasks or task categories.

Input / output format

Input: Question-answering prompts and options from the 33 tasks in the MosaicML evaluation gauntlet.

Output: Model-generated answer or selected option for each question.

Scoring recipe

def compute_avg_normalized_accuracy(task_accuracies, random_guess_accuracies):
    normalized_scores = []
    for acc, r_acc in zip(task_accuracies, random_guess_accuracies):
        norm_acc = (acc - r_acc) / (1 - r_acc)
        normalized_scores.append(norm_acc)
    return sum(normalized_scores) / len(normalized_scores)

Common pitfalls

• Using test-set perplexity on the original pretraining data as a proxy for downstream performance is misleading; pruned data often worsens test perplexity while improving downstream accuracy.
• Averaging raw accuracies across tasks without normalizing by each task's random guessing baseline will bias results toward tasks with higher chance accuracy.

Evidence (verbatim from paper)

We evaluate models on 33 different downstream question-answering tasks using the MosaicML evaluation gauntlet (MosaicML, 2023a). Before averaging the accuracy across tasks, we normalize each task by the baseline of random guessing . Specifically,we normalize the accuracy of each individual task as $a_n = rac{a_m - a_r}{1 - a_r}$ ,where $a_m$ is the accuracy of the model and $a_r$ is the expected accuracy of random guessing. We report the average normalized accuracy for each task category as well as the average normalized accuracy across all task categories.

Citation

@misc{ankner2024perplexed,
  title={Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models},
  author={Ankner et al. (2024)},
  year={2024},
  note={arXiv:2405.20541}
}

• arXiv: 2405.20541