Pre Training Validation Loss

pre-training-validation-loss

Benchmarking down-scaled (not so large) pre-trained language models — Aßenmacher et al. (2021) (arXiv:2105.04876, 2021)

What this evaluates

Evaluates the generalization capability of a language model during the pre-training phase by measuring the average cross-entropy loss on a held-out validation corpus. Lower values indicate that the model has better learned the underlying token distribution and converges more effectively under the given architectural and training configurations.

Datasets

• WikiText-103 — total ?; splits: train (-1), val (-1)

Metrics

• pre-training validation loss (primary) — range: other
  • Average cross-entropy loss computed over all tokens in the validation set during pre-training. Lower values indicate better predictive performance and faster convergence.

Input / output format

Input: Tokenized sequences from the pre-training corpus passed through the Transformer model.

Output: Next-token probability distributions used to compute cross-entropy loss.

Scoring recipe

loss = 0
count = 0
for batch in validation_data:
    logits = model(batch)
    loss += cross_entropy(logits, batch.targets)
    count += batch.num_tokens
return loss / count

Common pitfalls

• Model size is defined as non-embedding parameters rather than total parameters, which can cause confusion when comparing to standard scaling laws or other papers.
• The metric tracks pre-training convergence rather than downstream task performance or fine-tuning accuracy, so it should not be used to judge final task capability.

Evidence (verbatim from paper)

To examine convergence characteristics, we monitor the pre-training validation loss of several systems and test how this loss corresponds to different model sizes and shapes.

Citation

@misc{assenmacher2021benchmarking,
  title={Benchmarking down-scaled (not so large) pre-trained language models},
  author={Aßenmacher et al. (2021)},
  year={2021},
  note={arXiv:2105.04876}
}

• arXiv: 2105.04876