Spice Corpus Self Play

SPICE: Corpus-Grounded Self-Play for Reasoning

Self-play RL struggles with improvement plateaus because agents exhaust limited task spaces. SPICE grounds self-play in a real-world document corpus, enabling continuous challenge generation at the frontier of capability.

The corpus provides the rich, near-inexhaustible signal necessary for sustained reasoning improvement.

Core Concept

Two complementary roles:

• Challenger: mines difficult examples from corpus to create tasks
• Reasoner: solves Challenger-generated tasks

The Challenger creates automatic curriculum using corpus, addressing the fundamental limitation of closed-loop self-play: limited task diversity.

Architecture Overview

• Corpus of documents for task mining
• Challenger model: identifies difficult, solvable problems
• Reasoner model: attempts to solve mined problems
• Shared reward signal for co-training

Implementation Steps

Implement Challenger that mines tasks from corpus:

class ChallengerModel:
    def __init__(self, llm, corpus):
        self.llm = llm
        self.corpus = corpus

    def mine_task(self, reasoner_capability_level):
        """Generate task at boundary of reasoner capability."""
        # Sample documents from corpus
        candidates = self.corpus.sample(num_candidates=100)

        difficulty_scores = []
        for doc in candidates:
            # Extract problem from document
            problem = self.llm.extract_problem(doc)

            # Estimate difficulty relative to reasoner capability
            difficulty = self.estimate_difficulty(problem, reasoner_capability_level)

            difficulty_scores.append((problem, difficulty))

        # Select problems at frontier of capability
        # Not too easy (already solved), not impossible
        target_difficulty = reasoner_capability_level + 0.2
        frontier_problems = [
            p for p, d in difficulty_scores
            if abs(d - target_difficulty) < 0.1
        ]

        return frontier_problems[0] if frontier_problems else candidates[0]

    def estimate_difficulty(self, problem, current_capability):
        """Score problem difficulty."""
        # Use embedding similarity to past problems
        problem_emb = self.llm.embed(problem)

        # Find similar problems reasoner has solved
        similar_solved = []
        for solved_problem in self.reasoner_history:
            similarity = cosine_similarity(
                problem_emb,
                self.llm.embed(solved_problem)
            )
            similar_solved.append(similarity)

        # Difficulty = opposite of average similarity to solved problems
        avg_similarity = sum(similar_solved) / len(similar_solved) if similar_solved else 0
        return 1 - avg_similarity

Implement Reasoner training with corpus-generated tasks:

class ReasonerModel:
    def __init__(self, llm):
        self.llm = llm
        self.solved_problems = []

    def solve(self, task):
        """Attempt to solve mined task."""
        solution = self.llm.generate(task, max_tokens=500)
        return solution

    def get_reward(self, task, solution):
        """Verify solution quality."""
        # Use external verifier if available
        # E.g., Python execution, ground truth, heuristics
        return verify_solution(task, solution)

    def train_step(self, task, solution, reward):
        """RL update from corpus-generated task."""
        # Policy gradient update
        log_prob = self.llm.compute_log_prob(task, solution)
        loss = -log_prob * reward

        # Backward and update
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

        # Track solved problems for difficulty estimation
        if reward > threshold:
            self.solved_problems.append(task)

Implement the co-training loop:

def train_spice(challenger, reasoner, corpus, num_rounds=100):
    """Self-play loop with corpus grounding."""
    for round_idx in range(num_rounds):
        # Challenger mines task at frontier
        task = challenger.mine_task(reasoner.capability_level)

        # Reasoner attempts task
        solution = reasoner.solve(task)

        # Evaluate solution
        reward = reasoner.get_reward(task, solution)

        # Train both models
        reasoner.train_step(task, solution, reward)

        # Update Challenger's understanding of reasoner capability
        challenger.update_reasoner_profile(task, reward)

        if round_idx % 100 == 0:
            # Evaluate on held-out benchmarks
            benchmark_score = evaluate_on_benchmark(reasoner)
            print(f"Round {round_idx}: Benchmark = {benchmark_score:.2%}")

        # Corpus prevents task exhaustion
        # Challenger can always mine new problems

Practical Guidance

When to use:

• Reasoning task improvement without external data
• Scenarios with accessible document corpus
• Long-horizon training where diversity matters
• Scaling reasoning capabilities efficiently

When NOT to use:

• Tasks with no document corpus
• When explicit ground truth supervision exists (supervised better)
• Real-time systems (mining adds latency)

Common pitfalls:

• Corpus too small (task repetition, plateau)
• Difficulty estimation too loose (not at frontier)
• Not tracking solved problems (difficulty unchanged)
• Corpus containing solutions (shortcuts learning)

Reference: SPICE on arXiv