Pretraining Data Refinement

Improve pretraining efficiency by refining noisy data through expert-guided programs: learn to generate deletion operations that clean documents, achieving 2.6-7.2% performance gains with fewer training tokens.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 1e5eb00666

Frequently asked questions

npx get-skill add adu2021/pretraining-data-refinement