Overclocking LLM Reasoning

Overclocking LLM Reasoning

Core Concept

Reasoning models encode their progress through thinking phases in hidden representations. By extracting these "thinking progress vectors" (TPVs), you can monitor and manipulate reasoning length at inference time. The key insight: models maintain an internal estimate of their relative position within the explicit thinking phase, which can be extracted and controlled via intervention on hidden states.

Architecture Overview

• Progress monitoring: Train linear/GRU models to detect position in thinking phase
• Progress vector extraction: Identify TPVs from hidden state trajectories
• Intervention mechanism: Shift hidden representations along progress vectors to accelerate thinking
• Efficiency gains: Reduce token usage by 30% while improving answer quality

Implementation

Step 1: Train Progress Monitor

Build a model to predict progress position from hidden states:

class ProgressMonitor:
    def __init__(self, hidden_dim: int, model_type: str = "linear"):
        self.hidden_dim = hidden_dim
        self.model_type = model_type

        if model_type == "linear":
            self.monitor = torch.nn.Linear(hidden_dim, 1)
        else:  # GRU
            self.monitor = torch.nn.GRU(
                hidden_dim,
                64,
                batch_first=True
            )
            self.head = torch.nn.Linear(64, 1)

        self.optimizer = torch.optim.Adam(
            self.monitor.parameters(),
            lr=1e-4
        )

    def train_on_thinking_traces(self,
                                 hidden_states: torch.Tensor,
                                 token_positions: torch.Tensor,
                                 max_thinking_tokens: int):
        """Train monitor to predict normalized progress position."""

        # Normalize positions to [0, 1]
        normalized_positions = token_positions / max_thinking_tokens
        normalized_positions = torch.clamp(
            normalized_positions,
            0,
            1
        )

        # Predict progress for each hidden state
        if self.model_type == "linear":
            predictions = self.monitor(hidden_states)
        else:
            gru_out, _ = self.monitor(hidden_states)
            predictions = self.head(gru_out)

        # MSE loss on progress prediction
        loss = torch.nn.functional.mse_loss(
            predictions.squeeze(-1),
            normalized_positions
        )

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

        return loss.item()

    def predict_progress(self, hidden_state: torch.Tensor) -> float:
        """Predict current progress (0 to 1)."""
        if self.model_type == "linear":
            progress = self.monitor(hidden_state).item()
        else:
            gru_out, _ = self.monitor(hidden_state.unsqueeze(0))
            progress = self.head(gru_out).squeeze(-1).item()

        return torch.sigmoid(torch.tensor(progress)).item()

Step 2: Extract Thinking Progress Vectors

Identify direction of progress in hidden space:

class ThinkingProgressExtractor:
    def __init__(self, monitor: ProgressMonitor):
        self.monitor = monitor

    def extract_progress_vector(self,
                               hidden_trajectory: torch.Tensor
                               ) -> torch.Tensor:
        """Extract dominant direction of progress through thinking."""

        # Get progress predictions across trajectory
        progress_scores = []
        for h in hidden_trajectory:
            prog = self.monitor.predict_progress(h)
            progress_scores.append(prog)

        progress_scores = torch.tensor(progress_scores)

        # PCA to find primary direction of progress
        centered = hidden_trajectory - hidden_trajectory.mean(dim=0)
        U, S, V = torch.linalg.svd(centered, full_matrices=False)

        # First component represents progress direction
        progress_vector = V[0]  # Shape: (hidden_dim,)

        # Normalize by progress signal strength
        progress_signal = progress_scores.std()
        progress_vector = progress_vector * progress_signal

        return progress_vector

    def extract_batch_vectors(self,
                             batch_trajectories: list) -> torch.Tensor:
        """Extract progress vectors for a batch of thinking traces."""
        vectors = []

        for trajectory in batch_trajectories:
            vector = self.extract_progress_vector(trajectory)
            vectors.append(vector)

        return torch.stack(vectors)

Step 3: Implement Progress Vector Intervention

Manipulate hidden states to control thinking length:

class ThinkingIntervenor:
    def __init__(self, progress_vector: torch.Tensor):
        self.progress_vector = progress_vector
        self.progress_vector = (self.progress_vector /
                                self.progress_vector.norm())

    def intervene_hidden_state(self, hidden_state: torch.Tensor,
                              alpha: float) -> torch.Tensor:
        """Shift hidden state along progress vector.

        alpha > 0: accelerate thinking (move toward completion)
        alpha < 0: slow thinking (move away from completion)
        """
        return hidden_state + alpha * self.progress_vector

    def generate_with_intervention(self, model,
                                  prompt: str,
                                  max_thinking_tokens: int = 512,
                                  alpha: float = 1.0) -> str:
        """Generate response with progress vector intervention."""

        input_ids = model.tokenize(prompt)

        # Generate thinking phase
        thinking_output = model.generate(
            input_ids,
            max_new_tokens=max_thinking_tokens,
            output_hidden_states=True,
            return_dict_in_generate=True
        )

        # Get final hidden state from thinking phase
        thinking_hidden = thinking_output.hidden_states[-1][-1]

        # Apply intervention
        intervened_hidden = self.intervene_hidden_state(
            thinking_hidden,
            alpha
        )

        # Continue generation from intervened state
        response = model.continue_generation_from_hidden(
            intervened_hidden,
            max_new_tokens=256
        )

        return model.detokenize(response)

Step 4: Optimize Intervention Strength

Find optimal alpha for efficiency-quality tradeoff:

def find_optimal_intervention(model,
                             test_questions: list,
                             intervener: ThinkingIntervenor,
                             correct_answers: list) -> float:
    """Search for alpha that maximizes accuracy per token."""

    best_alpha = 0.0
    best_score = 0.0

    for alpha in [0.0, 0.5, 1.0, 1.5, 2.0]:
        correct = 0
        total_tokens = 0

        for question, answer in zip(test_questions, correct_answers):
            response = intervener.generate_with_intervention(
                model,
                question,
                alpha=alpha
            )

            if evaluate_correctness(response, answer):
                correct += 1

            total_tokens += count_tokens(response)

        accuracy = correct / len(test_questions)
        efficiency = accuracy / (total_tokens / len(test_questions))

        if efficiency > best_score:
            best_score = efficiency
            best_alpha = alpha

    return best_alpha

Practical Guidance

Progress Monitoring: Linear monitors work well for simple progress prediction. Use GRU-based monitors for longer thinking trajectories with more complex patterns.

Extraction Strategy: PCA on centered hidden states reveals the primary thinking progress direction. Multiple runs produce consistent vectors, indicating robust estimation.

Intervention Tuning: Start with alpha=1.0 (natural progress speed). Increase alpha for efficiency-focused tasks, decrease for complex reasoning requiring more exploration.

Performance Gains: On math problems with 512-token budgets, intervention achieves 80%+ improvements (54 correct vs 100 baseline), with 30% average token reduction.

When to Apply: Use when your reasoning model exhibits overthinking (allocates tokens late in phase without improving answers) or when token budgets are constrained.

Reference

The approach works because models maintain internal progress estimates that can be extracted and manipulated. Key insight: controlling thinking length via progress vector intervention outperforms temperature-based controls because it targets fundamental progress encoding rather than probability distributions.