Downstream Scaling Eval

downstream-scaling-eval

Language models scale reliably with over-training and on downstream tasks — Gadre et al. (2024) (arXiv:2403.08540, 2024)

What this evaluates

This evaluation probes how reliably language model scaling laws predict performance in over-trained regimes, where models are trained with significantly more tokens than parameters. It measures both next-token prediction accuracy on a held-out corpus and generalization across a broad suite of downstream zero-shot and few-shot tasks.

Datasets

• C4 eval — total ?; splits: eval (-1)
• LLM-foundry — total ?; splits: eval (-1)

Metrics

• Validation loss (primary) — range: other
  • Cross-entropy between the model's output distribution and the one-hot ground truth, averaged over all tokens in a sequence and across all sequences in the dataset.
• Average top-1 error — range: percent
  • Uniform average of the top-1 error (1 - accuracy) across 46 downstream tasks from LLM-foundry.
• Relative prediction error — range: other
  • |ζ(C,M) - ζ_GT| / ζ_GT, where ζ represents either the validation loss L or the average top-1 error Err.

Input / output format

Input: Text sequences for language modeling; prompts containing task instructions and optional few-shot examples for downstream evaluations.

Output: Next-token probability distributions for language modeling; discrete class labels or text completions for downstream tasks.

Scoring recipe

def compute_metrics(predictions, golds, metric_type):
    if metric_type == "validation_loss":
        total = sum(cross_entropy(p, g) for p, g in zip(predictions, golds))
        return total / len(golds)
    elif metric_type == "average_top_1_error":
        errors = []
        for task_preds, task_golds in zip(predictions, golds):
            acc = sum(1 for p, g in zip(task_preds, task_golds) if p == g) / len(task_golds)
            errors.append(1.0 - acc)
        return mean(errors)
    elif metric_type == "relative_prediction_error":
        pred_val, gt_val = predictions, golds
        return abs(pred_val - gt_val) / gt_val

Common pitfalls

• Models trained between 5.2×10^16 and 5.2×10^17 FLOPs are explicitly excluded from scaling law fitting because they over-perform due to receiving more optimization steps than neighboring configurations.
• Hyperparameters are tuned exclusively on the OpenLM eval validation set; no tuning is performed on downstream task validation sets or for token multipliers other than M=20.
• Runs that would require more tokens than available in a dataset (e.g., 0.411B model at M=640 on C4) are omitted, creating uneven compute coverage across datasets.

Evidence (verbatim from paper)

We consider three main metrics: (i) Validation loss, which is the cross entropy between a model’s output and the one-hot ground truth, averaged over all tokens in a sequence and over all sequences in a dataset. (ii) Average top-1 error, which is a uniform average over 46 downstream evaluations sourced from LLM-foundry. We also look at the mean top-1 error for the subset of 17 evaluations identified in the paragraph above. For a complete list of downstream evaluation datasets, see Appendix[D]. To measure how good a prediction ζ(C,M) is, we measure (iii) Relative prediction error: |ζ(C,M)-ζ_GT|/ζ_GT, where ζ is the loss L or the average top-1 error Err.

Citation

@misc{gadre2024scaling,
  title={Language models scale reliably with over-training and on downstream tasks},
  author={Gadre et al. (2024)},
  year={2024},
  note={arXiv:2403.08540}
}

• arXiv: 2403.08540