adu2021 SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis
Core Concept
SynthRL addresses data scarcity in visual reasoning tasks by automatically synthesizing challenging training questions from a seed dataset. Rather than manually creating thousands of visual math problems, the framework programmatically generates harder variants and verifies their correctness through automatic checkers.
The approach synthesizes 3,300+ additional challenging questions from approximately 8,000 seed samples, then trains vision-language models using reinforcement learning with verifiable reward signals. The key advantage is that synthesized data gains compound across test-time compute scaling, with improvements most pronounced on the hardest evaluation samples.
Architecture Overview
- Seed Question Selection: Identify representative problems from original dataset
- Augmentation Pipeline: Automatically generate harder variants through question transformation
- Verification Mechanism: Validate correctness of synthesized questions via automated checking
- RLVR Framework: Reinforcement learning with verifiable rewards (not learned rewards)
- Out-of-Domain Testing: Evaluate on benchmarks different from training distribution
- Scalable Data Generation: Generate thousands of new questions with minimal manual effort
Implementation
The following steps outline how to implement verifiable data synthesis for visual reasoning:
- Prepare seed dataset - Collect initial visual reasoning problems with verified answers
- Select candidates for augmentation - Identify problems suitable for difficulty increase
- Generate harder variants - Apply transformations to create more challenging versions
- Verify correctness - Automatically check that generated questions are correct and harder
- Prepare RL training - Create question-answer pairs with verifiable reward signals
- Train with RL - Optimize model using reinforcement learning on synthetic data
- Evaluate on benchmarks - Test on out-of-domain visual reasoning tasks
from typing import List, Dict, Tuple, Optional
import torch
class QuestionSynthesizer:
"""Generate harder variants of visual reasoning questions."""
def __init__(self, base_model, transformer_templates: List[str]):
self.model = base_model
self.templates = transformer_templates
def select_candidates(self, seed_questions: List[Dict], selection_ratio: float = 0.3) -> List[Dict]:
"""Select representative questions for augmentation."""
num_select = int(len(seed_questions) * selection_ratio)
# Simple heuristic: select diverse difficulty range
sorted_by_difficulty = sorted(seed_questions,
key=lambda x: x.get('difficulty', 0.5))
candidates = (sorted_by_difficulty[:num_select//2] +
sorted_by_difficulty[-num_select//2:])
return candidates
def augment_question(self, question: Dict, transformation: str) -> Optional[Dict]:
"""Apply transformation to make question harder."""
original_text = question["question"]
image = question["image"]
answer = question["answer"]
# Example transformations for visual math problems
augmentations = {
"multi_step": f"First calculate intermediate result. {original_text}",
"constraints": f"With the constraint that all values are positive: {original_text}",
"complexity": f"This problem involves more complex relationships: {original_text}",
"precision": f"Provide answer accurate to 3 decimal places: {original_text}"
}
augmented_text = augmentations.get(transformation, original_text)
return {
"question": augmented_text,
"image": image,
"original_answer": answer,
"transformation": transformation
}
def generate_synthetic_dataset(self, seed_data: List[Dict],
augmentations_per_question: int = 4) -> List[Dict]:
"""Generate synthetic questions from seed data."""
candidates = self.select_candidates(seed_data)
synthetic = []
for question in candidates:
for i, template in enumerate(self.templates[:augmentations_per_question]):
aug_question = self.augment_question(question, template)
if aug_question:
synthetic.append(aug_question)
return synthetic
class QuestionVerifier:
"""Verify correctness and difficulty of synthetic questions."""
def __init__(self, reference_solver):
self.solver = reference_solver
def verify_correctness(self, question: Dict) -> Tuple[bool, Optional[float]]:
"""Check if synthesized question has correct answer."""
try:
predicted_answer = self.solver.solve(question["image"], question["question"])
expected_answer = question.get("original_answer")
# Flexible comparison (exact match or numerical tolerance)
is_correct = self._compare_answers(predicted_answer, expected_answer)
confidence = self._estimate_confidence(predicted_answer)
return is_correct, confidence
except Exception as e:
return False, None
def verify_difficulty(self, original_question: Dict, synthetic_question: Dict) -> float:
"""Assess if synthetic question is harder than original."""
# In practice, use model predictions or heuristics
original_str = original_question["question"]
synthetic_str = synthetic_question["question"]
# Simple heuristic: length increase suggests complexity
length_ratio = len(synthetic_str.split()) / len(original_str.split())
return min(1.0, length_ratio)
def _compare_answers(self, predicted, expected) -> bool:
"""Compare answers with tolerance for numerical problems."""
if isinstance(predicted, (int, float)) and isinstance(expected, (int, float)):
return abs(predicted - expected) < 0.01
return str(predicted).strip() == str(expected).strip()
def _estimate_confidence(self, answer) -> float:
"""Estimate confidence in answer (simplified)."""
return 0.95 if answer else 0.1
def filter_synthetic_data(self, synthetic_questions: List[Dict],
quality_threshold: float = 0.9) -> Tuple[List[Dict], Dict]:
"""Filter synthetic questions by quality and difficulty."""
valid_questions = []
stats = {"total": len(synthetic_questions), "valid": 0, "rejected": 0, "avg_difficulty": 0}
difficulties = []
for question in synthetic_questions:
is_correct, _ = self.verify_correctness(question)
if not is_correct:
stats["rejected"] += 1
continue
# Verify it's actually harder
original = question.get("_original", {})
difficulty_gain = self.verify_difficulty(original, question)
if difficulty_gain > 0.0: # Any difficulty increase is acceptable
valid_questions.append(question)
difficulties.append(difficulty_gain)
stats["valid"] += 1
stats["avg_difficulty"] = sum(difficulties) / len(difficulties) if difficulties else 0
return valid_questions, stats
class VisualReasoningRLTrainer:
"""Train vision-language model with reinforcement learning on synthetic data."""
def __init__(self, model, verifier: QuestionVerifier):
self.model = model
self.verifier = verifier
def compute_reward(self, question: Dict, predicted_answer: str) -> float:
"""Compute verifiable reward signal."""
is_correct, confidence = self.verifier.verify_correctness(question)
# Reward is high if correct and confident
if is_correct:
return confidence if confidence else 0.95
else:
return 0.0 # No partial credit for incorrect answers
def rl_train_step(self, batch_questions: List[Dict], optimizer,
discount_factor: float = 0.99) -> float:
"""Execute one RL training step."""
total_loss = 0.0
for question in batch_questions:
# Generate prediction
predicted = self.model.generate(question["image"], question["question"])
# Compute reward
reward = self.compute_reward(question, predicted)
# RL loss: maximize expected reward
# Simplified: cross-entropy loss weighted by reward
loss = -torch.tensor(reward, dtype=torch.float32)
total_loss += loss.item()
# Backpropagation (simplified)
optimizer.zero_grad()
loss.backward()
optimizer.step()
return total_loss / len(batch_questions)
def train(self, synthetic_dataset: List[Dict], num_epochs: int = 3,
batch_size: int = 32, learning_rate: float = 1e-4) -> Dict:
"""Train model on synthetic dataset."""
optimizer = torch.optim.Adam(self.model.parameters(), lr=learning_rate)
metrics = {"epoch_losses": [], "total_correct": 0, "total_samples": 0}
for epoch in range(num_epochs):
epoch_loss = 0.0
for i in range(0, len(synthetic_dataset), batch_size):
batch = synthetic_dataset[i:i+batch_size]
loss = self.rl_train_step(batch, optimizer)
epoch_loss += loss
avg_loss = epoch_loss / (len(synthetic_dataset) // batch_size)
metrics["epoch_losses"].append(avg_loss)
print(f"Epoch {epoch}: Loss = {avg_loss:.4f}")
return metrics
class SynthRLPipeline:
"""End-to-end pipeline for synthetic data generation and RL training."""
def __init__(self, base_model, reference_solver, templates: List[str]):
self.synthesizer = QuestionSynthesizer(base_model, templates)
self.verifier = QuestionVerifier(reference_solver)
self.trainer = VisualReasoningRLTrainer(base_model, self.verifier)
def run_pipeline(self, seed_data: List[Dict], num_epochs: int = 3) -> Dict:
"""Run full synthesis and training pipeline."""
print("Step 1: Generating synthetic questions...")
synthetic = self.synthesizer.generate_synthetic_dataset(seed_data)
print("Step 2: Verifying synthetic data quality...")
valid_synthetic, stats = self.verifier.filter_synthetic_data(synthetic)
print(f" Validation: {stats['valid']}/{stats['total']} valid ({100*stats['valid']/stats['total']:.1f}%)")
print("Step 3: Training with RL...")
metrics = self.trainer.train(valid_synthetic, num_epochs=num_epochs)
return {
"synthesis_stats": stats,
"training_metrics": metrics,
"synthetic_dataset_size": len(valid_synthetic)
}
Practical Guidance
Data synthesis configuration:
- Augmentations per question: 3-5 variants per seed question; balance diversity with verification cost
- Selection ratio: 30-50% of seed data; focus on representative examples
- Quality threshold: 0.85-0.95; higher threshold ensures clean training data
Verification strategy:
- Reference solver: Use symbolic solver for math, multiple models for consensus checking
- Difficulty assessment: Measure by problem complexity metrics, not length alone
- Rejection rate: Expect 20-40% of initial synthetic questions to fail verification
RL training setup:
- Batch size: 32-64 depending on dataset size and GPU memory
- Learning rate: 1e-4 to 1e-5 for stable training on noisy RL signals
- Epochs: 2-5 epochs typically sufficient; monitor for overfitting
When to use:
- Scaling training data for visual reasoning without manual labeling
- Tasks with verifiable correct answers (math, logic, code)
- Improving out-of-domain generalization through diverse synthetic data
- Research on automated curriculum learning and data augmentation
When NOT to use:
- Tasks without automated verification (open-ended reasoning, creative tasks)
- Domains where synthetic data distribution differs significantly from real data
- Systems requiring exact distribution matching (GANs, etc.)
- Real-time applications where verification overhead is prohibitive
Common pitfalls:
- Distribution shift: Synthetic data may not match test distribution; validate on original benchmarks
- Verification bias: Incorrect reference solver validates wrong answers; use multiple verifiers
- Difficulty plateau: Transformations may not consistently increase difficulty; use adaptive sampling
- Mode collapse: RL training may overfit to verification artifacts; regularize with original data
- Computational cost: Verification adds significant overhead; batch verification to amortize cost
Reference
SynthRL synthesizes 3,300+ additional challenging questions from approximately 8,000 seed samples, demonstrating consistent improvements across five visual math reasoning benchmarks. Gains are most pronounced on the hardest evaluation samples, suggesting the approach effectively elicits deeper reasoning patterns.
Original paper: "SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis" (arxiv.org/abs/2506.02096)