Reasoning Palette

Overview

Reasoning Palette addresses a fundamental inefficiency in LLM reinforcement learning: standard sampling schemes create limited trajectory diversity because token-level randomness doesn't generate sufficiently different reasoning approaches. This framework shifts exploration to a learned latent space of reasoning strategies.

Core Technique

The key insight is that reasoning diversity should come from high-level strategy changes, not token-level noise.

VAE-Based Latent Strategy Space: Learn a continuous space encoding diverse reasoning patterns from multi-domain data.

# VAE for reasoning strategy encoding
class ReasoningStrategyVAE:
    def __init__(self, latent_dim=16):
        self.encoder = Encoder()  # Question-answer → latent
        self.decoder = Decoder()  # Latent → prefix embedding
        self.latent_dim = latent_dim

    def encode_strategies(self, qa_pairs):
        """
        Encode diverse reasoning patterns from mathematical,
        coding, and QA domains into shared latent space.
        """
        strategies = []
        for question, answer in qa_pairs:
            # Embed both question and answer
            q_embed = embedding_model(question)
            a_embed = embedding_model(answer)

            # Mean-pool to get strategy representation
            strategy_rep = (q_embed + a_embed) / 2

            # Encode into latent space
            latent = self.encoder(strategy_rep)
            strategies.append(latent)

        return torch.stack(strategies)

    def sample_and_decode(self):
        """
        Sample from Gaussian latent space and decode to
        prefix embeddings that steer reasoning.
        """
        # Sample from standard normal
        z = torch.randn(1, self.latent_dim)

        # Decode to prefix embedding
        prefix_embedding = self.decoder(z)

        return prefix_embedding

Prefix Conditioning for Reasoning Steering: Decoded latent vectors become learnable token prefixes prepended to prompts.

def apply_strategy_prefix(model, question, latent_sample):
    """
    Prepend latent-decoded prefix to question.
    Prefix steers the model's internal reasoning before generation.
    """
    # Decode latent to prefix embedding
    prefix_embedding = vae_decoder(latent_sample)

    # Prepend to question tokens
    prompt_with_prefix = [prefix_embedding] + tokenize(question)

    # Forward pass (prefix modulates reasoning trajectory)
    output = model.generate(prompt_with_prefix)

    return output

Scheduled Integration in RL Training: During RL, latent-guided prefixes enable structured exploration that transitions from exploration to exploitation.

def rl_training_with_latent_guidance(model, dataset, epochs=10):
    """
    RL training with scheduled latent-prefix guidance.
    Early epochs: wide exploration via diverse latents
    Late epochs: exploit best strategies
    """
    for epoch in range(epochs):
        # Compute exploration schedule
        exploration_factor = 1.0 - (epoch / epochs)  # Decrease over time

        for batch in dataset:
            # Sample latent strategy
            if random.random() < exploration_factor:
                # Exploration phase: sample diverse latents
                latent = vae.sample_latent()
            else:
                # Exploitation phase: use best latent
                latent = best_latent_so_far

            # Generate with strategy prefix
            prefix = vae_decoder(latent)
            trajectory = model.generate_with_prefix(batch, prefix)

            # RL update
            reward = compute_reward(trajectory)
            model.update_via_grpo(trajectory, reward)

            # Optional: update best latent
            if reward > best_reward:
                best_latent_so_far = latent
                best_reward = reward

When to Use This Technique

Use Reasoning Palette when:

• RL training for reasoning tasks shows poor trajectory diversity
• Token-level randomness creates insufficient strategy variation
• Multi-domain reasoning (math, coding, QA) training
• Interpretability of reasoning strategies is valuable

When NOT to Use This Technique

Avoid this approach if:

• Single-domain or narrow reasoning required (simpler methods suffice)
• Computational overhead of VAE training unacceptable
• Token-level randomness adequately covers strategy space
• Real-time online learning requires immediate strategy adaptation

Implementation Notes

The framework requires:

• VAE training on diverse question-answer pairs
• Integration of prefix embeddings into model architecture
• RL training loop with scheduled exploration
• Supervised fine-tuning warm-up to sensitize model to prefixes

Key Performance

• Consistent improvements on mathematical reasoning benchmarks
• More diverse and explorative trajectories than token-level sampling
• Interpretable strategy representations
• Controllable reasoning behavior

References

• VAE-based latent space for reasoning strategies
• Prefix conditioning for internal reasoning modulation
• Scheduled RL integration for exploration-exploitation
• Multi-domain reasoning pattern learning