C4 Loss Scaling Eval

c4-loss-scaling-eval

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — Shengding Hu et al. (2024) (arXiv:2404.06395, 2024)

What this evaluates

Evaluates how optimal batch size and learning rate scale with model size and target loss during pre-training. It probes the stability of hyperparameters across different model scales and quantifies the relationship between batch size and validation loss on a standard text corpus.

Datasets

• C4 — total ?; splits: test (-1); HF allenai/c4

Metrics

• C4 Loss (primary) — range: [0, ∞)
  • Token-level cross-entropy loss computed over the C4 corpus. The paper uses this loss to derive a scaling law for optimal batch size: bs = 1.21e9 / L^6.24, where L is the C4 Loss.

Input / output format

Input: Tokenized text sequences sampled from the C4 dataset.

Output: Next-token probability distributions (logits) over the vocabulary.

Scoring recipe

def compute_c4_loss(model, dataloader):
    total_loss = 0.0
    total_tokens = 0
    model.eval()
    with torch.no_grad():
        for batch in dataloader:
            input_ids = batch['input_ids']
            labels = input_ids.clone()
            outputs = model(input_ids, labels=labels)
            loss = outputs.loss
            total_loss += loss.item() * labels.numel()
            total_tokens += labels.numel()
    return total_loss / total_tokens

Common pitfalls

• Assuming the optimal batch size is constant across model scales; the paper demonstrates it scales inversely with loss raised to a power (~6.24).
• The batch size formula relies on a rough loss prediction that is only available after training, making it a retrospective scaling guideline rather than a pre-training hyperparameter.

Evidence (verbatim from paper)

We observe the trend of the optimal batch size with loss on the C4 (Raffel et al., 2019) dataset (red line in the Figure 1). We use parabolas to fit the equal-loss points and connect the minima of the parabolas with red lines. The lines demonstrate the optimal batch size shifts large as the loss decreases. We then connect the three lines (see Figure 2) and find that the lines connect each other well into a linear relationship in the log space, from which we obtain the following relationship between batch size $bs$ and C4 Loss $L$ : $bs = rac{1.21 imes 10^9}{L^{6.24}}$ .

Citation

@misc{hu2024minicpm,
  title={MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies},
  author={Shengding Hu et al. (2024)},
  year={2024},
  note={arXiv:2404.06395}
}

• arXiv: 2404.06395