Held Out Test Loss

held-out-test-loss

Scaling Data-Constrained Language Models — Muennighoff et al. (2023) (arXiv:2305.16264, 2023)

What this evaluates

Evaluates language model generalization and overfitting by measuring cross-entropy loss on a held-out test set. It probes how well the model retains predictive performance when trained on repeated or constrained data subsets.

Datasets

• C4 — total ?; splits: test (-1)

Metrics

• held-out test loss (primary) — range: other
  • Average cross-entropy loss computed over a held-out test set drawn from the same underlying dataset as the training data. Lower values indicate better generalization.

Input / output format

Input: Tokenized text sequences from the C4 dataset.

Output: Predicted probability distribution over the next token.

Scoring recipe

loss = 0.0
for tokens in test_set:
    logits = model(tokens)
    loss += cross_entropy(logits, tokens)
return loss / len(test_set)

Common pitfalls

• Using training loss instead of held-out test loss can mask severe overfitting when data is repeated across epochs.
• Failing to shuffle data between epochs or ensuring maximal overlap between training subsets can bias the evaluation of data-constrained scaling.

Evidence (verbatim from paper)

we report loss on a held-out test set unless otherwise specified (see Appendix K). This contrasts training loss used in [42], but should not alter our findings as the held-out data stems from the same underlying dataset.

Citation

@misc{muennighoff2023scalingdata,
  title={Scaling Data-Constrained Language Models},
  author={Muennighoff et al. (2023)},
  year={2023},
  note={arXiv:2305.16264}
}

• arXiv: 2305.16264