Scaling Laws Optimal Data Mixtures

Predict optimal data mixture proportions for multi-domain LLM training using scaling laws that require only 10-20 small experiments. Determine which domains should contribute how much data across model sizes (186M-7B), reducing computational waste in foundation model pretraining.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit e930a1e1df

Frequently asked questions

npx get-skill add adu2021/scaling-laws-optimal-data-mixtures