Pretrain Zero Active Pretraining

Bilevel min-max optimization where mask generator selects informative spans from pretraining data and mask predictor recovers them via chain-of-thought, enabling effective RL pretraining on noisy corpora without supervised fine-tuning.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 17edd3b457

Frequently asked questions

npx get-skill add adu2021/pretrain-zero-active-pretraining