Sparse Critical Rlvr Token Analysis

Analyzes how Reinforcement Learning from Verification Rewards (RLVR) improves reasoning by examining token-level probability distributions. Finds that >83% of token positions exhibit near-zero divergence—RL operates through sparse, targeted refinements. Cross-sampling experiments show 1.5-7.8% RL-selected tokens recover full gains, while reverting 5-10% of RL tokens collapses performance. Reveals that RL primarily reallocates probability within existing candidates (80% overlap in top-k tokens), not inventing novel tokens. Trigger: When analyzing LLM reasoning improvements, apply token-level divergence analysis and cross-sampling to identify which positions drive gains and whether changes are sparse or distributed.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit bfceee09bf

Frequently asked questions

npx get-skill add adu2021/sparse-critical-rlvr-token-analysis