Vespo Variational Sequence Rl

Stabilize off-policy RL training for LLMs by deriving principled importance weight reshaping from variational optimization. Instead of heuristic clamping, VESPO uses closed-form exponential weighting W^c1 * exp(c2*(1-W)) to suppress overweighted samples while maintaining smooth gradients. Enables stable training at 64× policy staleness and under fully asynchronous execution with sequence-level operations that avoid length-dependent biases.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit 7431f5cb73

Frequently asked questions

npx get-skill add adu2021/vespo-variational-sequence-rl