Rlp Reinforcement Pretraining Objective

Improve reasoning during pretraining (not just post-training) by computing rewards from information gain—how much reasoning improves log-likelihood of observed tokens. Works at 1T token scale across diverse corpora.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit a43fdee767

Frequently asked questions

npx get-skill add adu2021/rlp-reinforcement-pretraining-objective