Rlp Reinforcement Pretraining Objective

RLP: Reinforcement as Pretraining Objective

RLP introduces verifier-free reinforcement during pretraining itself. Rather than treating reasoning as post-training only, the approach computes dense reward signals from information gain: how much sampled chain-of-thought improves prediction of the next observed token. This enables reasoning to scale with pretraining data itself.

Core Architecture

• Information gain reward: No external verifier; reward = ΔLogLikelihood
• Reasoning-before-prediction: Sample CoT tokens, then next-token prediction
• Verifier-free: Dense rewards from data signal itself
• Efficient training: Runs on standard pretraining infrastructure
• Scale: Tested at 1T tokens with multiple model sizes

Implementation Steps

Setup reinforcement pretraining objective:

# Initialize RLP pretraining trainer
from rlp_pretrain import ReasoningPretrainer, InformationGainReward

pretrainer = ReasoningPretrainer(
    model_config="your_transformer_config",
    reasoning_tokens_per_position=G,  # G samples per position
    reasoning_max_length=2048
)

# Define information gain reward
reward_function = InformationGainReward(
    use_ema_teacher=True,
    ema_decay=0.999,  # EMA snapshot for baseline
    importance_sampling=True
)

# Initialize with standard pretraining setup
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
scaler = torch.cuda.amp.GradScaler()

Execute reinforcement pretraining:

# Training loop with reasoning and next-token prediction
for step, batch in enumerate(pretraining_dataloader):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]

    with torch.amp.autocast(device_type="cuda"):
        # Standard forward pass
        logits = model(input_ids, attention_mask)

        # For each position, sample reasoning before prediction
        reasoning_rewards = []

        for pos in range(input_ids.shape[1] - 1):
            # Get predicted token at position pos
            next_token_id = input_ids[:, pos + 1]

            # Sample G reasoning trajectories
            # (In practice, use conditional sampling from model)
            reasoning_samples = model.sample_reasoning(
                context=input_ids[:, :pos],
                num_samples=G,  # typically G=16
                max_length=min(2048, max_reasoning_length)
            )

            # For each reasoning trajectory, compute conditional probability
            log_probs_with_reasoning = []
            for reasoning in reasoning_samples:
                # Concatenate reasoning + input to position pos
                augmented_context = torch.cat(
                    [input_ids[:, :pos], reasoning],
                    dim=1
                )

                # Predict next token probability with reasoning
                augmented_logits = model(augmented_context, attention_mask=None)
                log_prob = F.log_softmax(augmented_logits[:, -1], dim=-1)[next_token_id]
                log_probs_with_reasoning.append(log_prob)

            # Compute information gain for each reasoning trajectory
            baseline_log_prob = F.log_softmax(logits[:, pos], dim=-1)[next_token_id]

            for log_prob_with_reasoning in log_probs_with_reasoning:
                information_gain = log_prob_with_reasoning - baseline_log_prob
                reasoning_rewards.append(information_gain)

        # Aggregate rewards for policy update
        reasoning_rewards = torch.stack(reasoning_rewards)

        # RLVR loss (policy gradient with baseline)
        log_probs_reasoning = model.compute_log_probs(reasoning_samples)
        rl_loss = -(log_probs_reasoning * (reasoning_rewards - reward_function.baseline)).mean()

        # Standard next-token prediction loss
        ntp_loss = F.cross_entropy(
            logits[:, :-1].reshape(-1, logits.shape[-1]),
            input_ids[:, 1:].reshape(-1)
        )

        # Combined loss
        total_loss = ntp_loss + rl_loss_weight * rl_loss

    # Backward pass
    scaler.scale(total_loss).backward()
    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()

    # Update EMA teacher for reward baseline
    reward_function.update_ema_teacher(model)

Practical Guidance

When to use RLP:

• Building foundation models where reasoning is important
• Scenarios where pretraining data naturally contains reasoning (code, math, papers)
• Settings where post-training compute budget limited
• Models targeting reasoning-heavy downstream tasks

When NOT to use:

• Domains where reasoning not naturally in data (image, audio pretraining)
• Purely autoregressive models where reasoning overhead excessive
• Real-time pretraining requiring minimal computational overhead
• Models where post-training RL already sufficient

Hyperparameters:

• G (num reasoning samples, 16): Tradeoff between quality and compute. Test 4-32 range
• Reasoning max length (2048): Allow ample reasoning space; reduce to 512 for speed
• RL loss weight: Start at 0.1; increase to 0.3-0.5 if model relies on reasoning
• EMA decay (0.999): Teacher staleness control; increase to 0.9999 for stability
• Importance sampling: Enable for large batch sizes; disable for memory constraints
• Completion length (2048): Max length for model reasoning; match to pretraining context

Data Composition Effects

Performance varies by corpus type:

• Code/math corpora: +8-12% improvements (reasoning naturally supported)
• Academic papers: +5-8% improvements (structured reasoning present)
• Web crawl: +2-4% improvements (less inherent reasoning)
• General text: +1-3% improvements (minimal reasoning signals)

Computational Characteristics

• Per-token overhead: G forward passes for reasoning sampling (G=16 → 16x)
• Mitigation: Amortize over batches; use gradient accumulation
• Effective cost: ~2-3x pretraining cost (with optimizations)
• Benefit: Reasoning capability without separate post-training phase

Generalization Properties

Trained models show strong cross-domain generalization:

• In-distribution: +5-8% gains on matching task types
• Out-of-distribution: +3-5% gains on unseen domains
• Scale: Benefits persist from 1.7B to 12B parameters
• Architecture robustness: Works with Transformers and Mamba-Transformers

Information Gain as Reward Signal

Key insight: next-token probability improvement quantifies reasoning quality:

• High information gain: Reasoning contributes to data likelihood
• Low information gain: Reasoning irrelevant or noisy
• Dense signal: Every position provides reward feedback
• Verifier-free: No external correctness model needed

Baseline Management

EMA teacher prevents distribution shift during training:

• Detached targets: No gradient flow through baseline
• Exponential averaging: Smooth updates prevent instability
• Architecture matching: Same model for baseline and policy

References

Builds on pretraining objectives, policy gradients, and information-theoretic reward design.