Spiral Zero Sum Game Reasoning

SPIRAL: Reasoning Through Competitive Self-Play on Zero-Sum Games

Language models trained with supervised learning struggle to develop genuinely sophisticated reasoning—they memorize patterns rather than learning to think through problems. Self-play on competitive games creates natural incentives for reasoning: a model must explain its strategy to convince itself (as the opponent) while also developing robust plans to win. SPIRAL demonstrates that this competitive framework outperforms supervised fine-tuning while requiring no human-annotated reasoning traces.

The core insight is that zero-sum games create a virtuous cycle: a stronger reasoning model becomes a tougher opponent, forcing continuous improvement. Unlike supervised learning where training data is fixed, self-play opponents dynamically evolve, preventing the model from exploiting dataset shortcuts and forcing genuine strategic thinking.

Core Concept

SPIRAL replaces supervised fine-tuning with a distributed self-play system where a single language model policy plays both sides of multiple two-player games, conditioned on player identity. The games are designed to require different reasoning skills—spatial reasoning, probabilistic inference, strategic negotiation—that transfer to academic benchmarks.

The key innovation is Role-Conditioned Advantage Estimation (RAE), which prevents "thinking collapse" where models abandon reasoning traces. RAE normalizes rewards relative to each player's expected performance, ensuring both players maintain interpretable chain-of-thought reasoning even as they improve.

Architecture Overview

• Distributed Actor-Learner System: Multiple actors generate trajectories through self-play; a central learner performs full-parameter updates with continuous opponent evolution
• Three Complementary Games: TicTacToe (spatial logic), Kuhn Poker (probabilistic reasoning), Simple Negotiation (strategic optimization)
• Role-Conditioned Advantage Estimation: Separate per-game, per-role baselines preventing reward variance from driving degenerate policies
• Multi-Game Training: Sequential or joint training across games with analysis of skill transfer
• Continuous Policy Updates: Both roles benefit from training improvements rather than fixed opponent strategies

Implementation

This implementation demonstrates the self-play training loop with role conditioning:

import torch
import torch.nn as nn
from typing import Tuple, Dict

class RoleConditionedPolicy(nn.Module):
    """
    Language model policy conditioned on player role.
    Same weights for both players, but input embedding encodes role.
    """
    def __init__(self, vocab_size=50000, hidden_dim=2048):
        super().__init__()
        self.embeddings = nn.Embedding(vocab_size, hidden_dim)
        self.role_embedding = nn.Embedding(2, hidden_dim)  # role 0 or 1
        self.transformer = TransformerCore(hidden_dim, depth=12)
        self.value_head = nn.Linear(hidden_dim, 1)

    def forward(self, game_state_tokens, role):
        """Generate action and evaluate position from given player perspective."""
        # Embed game state and condition on role
        state_embed = self.embeddings(game_state_tokens)
        role_embed = self.role_embedding(role)

        # Combine with broadcasting
        combined = state_embed + role_embed.unsqueeze(1)

        # Transformer processes combined state
        hidden = self.transformer(combined)

        # Value prediction from current player's perspective
        value = self.value_head(hidden[:, -1])
        return hidden, value


class RoleConditionedAdvantageEstimation:
    """
    RAE prevents thinking collapse by normalizing advantages per role.
    Each game-role pair maintains its own baseline.
    """
    def __init__(self, num_games=3, learning_rate=1e-3):
        # Separate baselines for each game and each role (0=player1, 1=player2)
        self.baselines = {}
        self.baseline_optimizers = {}

        for game_idx in range(num_games):
            for role in [0, 1]:
                key = (game_idx, role)
                baseline = nn.Linear(2048, 1)
                self.baselines[key] = baseline
                self.baseline_optimizers[key] = torch.optim.Adam(
                    baseline.parameters(), lr=learning_rate
                )

    def compute_advantage(self, returns, game_idx, role, hidden_states):
        """Compute advantage normalized per role in each game."""
        key = (game_idx, role)
        baseline = self.baselines[key]

        # Predict baseline (expected return for this role in this game)
        baseline_pred = baseline(hidden_states).squeeze(-1)

        # Advantage is deviation from role-specific baseline
        advantages = returns - baseline_pred.detach()

        # Update baseline to track true returns
        baseline_loss = torch.mean((baseline_pred - returns) ** 2)
        self.baseline_optimizers[key].zero_grad()
        baseline_loss.backward()
        self.baseline_optimizers[key].step()

        return advantages


def self_play_trajectory(
    policy: RoleConditionedPolicy,
    game_type: str,
    max_turns: int = 50
) -> Tuple[list, float, int]:
    """
    Generate a single self-play game where the same policy plays both roles.
    Returns trajectory (tokens, roles, rewards), final reward, and winner.
    """
    trajectory = []
    game_state = initialize_game(game_type)
    current_role = 0

    for turn in range(max_turns):
        # Current player generates action through policy
        state_tokens = encode_game_state(game_state)
        hidden, value = policy(state_tokens, role=current_role)

        # Sample action from model
        action = sample_action(hidden)

        # Store transition
        trajectory.append({
            'state_tokens': state_tokens,
            'role': current_role,
            'action': action,
            'value': value
        })

        # Execute action
        game_state, reward, done = game_state.apply_action(action)

        if done:
            break

        # Switch to other player
        current_role = 1 - current_role

    # Determine winner (reward from role 0 perspective)
    final_reward = game_state.get_reward_for_role(0)
    winner = 0 if final_reward > 0 else 1

    return trajectory, final_reward, winner


def train_step(
    policy: RoleConditionedPolicy,
    rae: RoleConditionedAdvantageEstimation,
    trajectories: list,
    game_idx: int,
    learning_rate: float = 5e-5
) -> float:
    """
    Perform policy gradient update using role-conditioned advantages.
    Prevents thinking collapse by normalizing per-role per-game.
    """
    optimizer = torch.optim.Adam(policy.parameters(), lr=learning_rate)
    total_loss = 0

    for trajectory, final_reward, winner in trajectories:
        # Compute returns (discounted cumulative rewards)
        returns = []
        cumulative = 0
        for step in reversed(range(len(trajectory))):
            # In final turn, get actual game reward; earlier turns get discounted future
            if step == len(trajectory) - 1:
                step_return = final_reward
            else:
                step_return = 0.99 * cumulative
            returns.insert(0, step_return)
            cumulative = step_return

        returns = torch.tensor(returns)

        # Compute advantages per role using RAE
        for step_idx, step_data in enumerate(trajectory):
            role = step_data['role']
            hidden = step_data['value'].unsqueeze(0)

            # Role-conditioned advantage prevents thinking collapse
            advantage = rae.compute_advantage(
                returns[step_idx:step_idx+1], game_idx, role, hidden
            )

            # Policy gradient: maximize log prob of winning action weighted by advantage
            log_prob = compute_log_prob(step_data)
            loss = -log_prob * advantage.detach()
            total_loss += loss

    # Update policy
    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()

    return total_loss.item()

Multi-game training coordination enables skill transfer:

def train_multi_game(policy, games=['tictactoe', 'kuhn_poker', 'negotiation'],
                     num_iterations=1000, batch_size=32):
    """
    Train on three games that develop complementary reasoning skills.
    Games transfer: spatial logic → probabilistic → strategic reasoning.
    """
    rae = RoleConditionedAdvantageEstimation(num_games=len(games))

    for iteration in range(num_iterations):
        for game_idx, game_type in enumerate(games):
            # Generate batch of self-play trajectories
            trajectories = [
                self_play_trajectory(policy, game_type)
                for _ in range(batch_size)
            ]

            # Update policy using role-conditioned advantages
            loss = train_step(policy, rae, trajectories, game_idx)

            if iteration % 100 == 0:
                print(f"Game {game_type}: Loss={loss:.4f}")

    return policy

Practical Guidance

When to use:

• Training reasoning capabilities without human annotation (self-play solves this)
• Developing models for competitive or strategic tasks (games train naturally)
• Creating robust thinking traces that persist through training (RAE prevents collapse)
• Transferring reasoning skills across domains (multi-game training)
• Benchmarking zero-shot chain-of-thought reasoning

When NOT to use:

• Single-task supervised learning where human labels are abundant and high-quality
• Real-time applications requiring strict inference time bounds (self-play creates large hidden states)
• Tasks without clear game-like objective structure (requires well-defined winning conditions)
• Environments where computational cost dominates (distributed training required)
• Problems where explicit step-by-step correctness matters more than final answer (games only reward outcomes)

Common pitfalls:

• Insufficient game diversity causing overfitting to specific game mechanics
• RAE baseline initialization too high, preventing early learning
• Fixed opponent strategies instead of continuous policy updates
• Ignoring per-role dynamics (both players need strong reasoning)
• Training imbalance across games when multi-game training
• Thinking collapse still occurring if RAE learning rate too high

Reference

"SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn RL", 2025. arxiv.org/abs/2506.24119