Golden Goose Task Synthesis

Golden Goose: Unlimited Task Synthesis from Text

Problem

Models trained with Reinforcement Learning from Verifiable Rewards (RLVR) saturate quickly on limited, manually-created datasets. Performance plateaus prevent continued improvement on reasoning tasks like mathematics and coding, where verifiable correct/incorrect answers are expensive to generate at scale.

The core challenge is that RLVR requires ground-truth labels for verification—yet most internet text (textbooks, forum posts, research papers) cannot be automatically verified. Traditional RLHF approaches require expensive human feedback, creating a scaling bottleneck.

Core Concept

Golden Goose converts unverifiable reasoning text into verifiable multiple-choice training problems through fill-in-the-middle masking. The key insight is that intermediate reasoning steps naturally form correct answers, while other plausible continuations become high-quality distractors.

The process extracts reasoning-rich passages, masks crucial internal steps, generates diverse alternatives, and filters problems by difficulty. This transforms passive text into active verification tasks without human intervention.

Architecture Overview

• Passage Identification: Scan corpora (textbooks, AoPS forums, code repositories) for reasoning-rich content with clear intermediate steps
• Masking Strategy: Identify contiguous spans of crucial reasoning and replace with [MASK] token, preserving the underlying problem structure
• Choice Generation: Retain masked content as ground-truth answer; generate 8 diverse distractors using language model sampling
• Difficulty Filtering: Score by self-consistency; remove problems scoring <0.2 or >0.8 confidence to focus on medium-difficulty problems
• Scale: Results in 700K+ verifiable problems suitable for standard RL training

Implementation

Step 1: Extract Reasoning Passages

Identify passages containing multi-step reasoning by scanning educational texts for sufficient length and step diversity.

import re

def extract_reasoning_passages(documents, min_steps=3, min_length=500):
    """Extract passages likely to contain multiple reasoning steps."""
    passages = []
    for doc in documents:
        # Look for patterns with numbered steps, clear logic flow
        if len(doc) > min_length and doc.count('\n') >= min_steps:
            passages.append(doc)
    return passages

Step 2: Create Fill-in-the-Middle Format

Select a contiguous span of crucial reasoning, remove it, and treat as the ground-truth answer.

def create_masked_problem(passage, model):
    """Convert reasoning passage to masked problem format."""
    # Identify crucial reasoning spans using language model analysis
    important_spans = model.identify_reasoning_spans(passage)

    # Select one span for masking
    span = important_spans[0]
    start, end = span['start'], span['end']

    # Create problem with [MASK]
    problem = passage[:start] + "[MASK]" + passage[end:]
    correct_answer = passage[start:end]

    return problem, correct_answer

Step 3: Generate Diverse Distractors

Sample plausible alternatives from the language model and filter for diversity and plausibility.

def generate_distractors(problem, correct_answer, model, num_distractors=8):
    """Generate diverse, plausible incorrect alternatives."""
    distractors = []

    # Sample continuations conditioned on the problem
    candidates = model.sample_continuations(
        problem,
        num_samples=50,
        temperature=0.8
    )

    # Filter for diversity and quality
    for candidate in candidates:
        if candidate != correct_answer:
            # Check for similarity to existing choices
            if not is_too_similar(candidate, [correct_answer] + distractors):
                distractors.append(candidate)
                if len(distractors) == num_distractors:
                    break

    return distractors

def is_too_similar(text1, candidates, threshold=0.85):
    """Check if text is too similar to existing candidates."""
    for candidate in candidates:
        similarity = compute_similarity(text1, candidate)
        if similarity > threshold:
            return True
    return False

Step 4: Filter by Difficulty

Remove problems that are too easy or too hard to avoid wasting training capacity.

def filter_by_difficulty(problem, choices, model, target_confidence=0.5, tolerance=0.3):
    """Filter problems by difficulty using self-consistency."""
    # Generate multiple solution attempts
    attempts = []
    for _ in range(5):
        response = model.solve_problem(problem)
        attempts.append(response)

    # Calculate confidence as fraction selecting correct answer
    correct_count = sum(1 for a in attempts if choices[0] in a)
    confidence = correct_count / len(attempts)

    # Keep problems with intermediate difficulty
    return abs(confidence - target_confidence) < tolerance

Practical Guidance

Hyperparameter Configuration

When to Use

• Continuous RLVR training: Overcome saturation on fixed datasets by generating new problems daily
• Scaling data: Replace expensive human annotation with automated task synthesis
• Domain adaptation: Bootstrap training data for new domains using domain-specific text corpora
• Benchmark construction: Create diverse evaluation sets from internet text

When Not to Use

• When human-verified problems are already abundant and inexpensive
• For tasks requiring ground-truth beyond binary correct/incorrect
• When masked passages don't naturally correspond to meaningful reasoning steps
• For open-ended reasoning where multiple valid paths exist equally

Common Pitfalls

1. Insufficient masking diversity: Selecting only the final step leaves context too obvious. Mask multiple reasoning styles to increase difficulty.
2. Distractor quality: Distractors that are obviously wrong or semantically identical reduce learning signal. Require both plausibility and diversity.
3. Difficulty bias: Filtering threshold must match your model's capability. Too tight filtering wastes data; too loose includes only trivial problems.
4. Domain mismatch: Text corpora from one domain may not generalize. Validate on target domain benchmarks.

Reference

Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text https://arxiv.org/abs/2601.22975