adu2021 Overview
SPARK eliminates the need for ground-truth step-level annotations by training process reward models (PRMs) using synthetic verification data. A generator produces diverse solutions while a verifier evaluates them, providing supervision that outperforms ground-truth-based training when used to train generative PRMs.
When to Use
- Training process reward models without expensive step annotations
- Mathematical reasoning and problem-solving tasks
- Scenarios where verification is cheaper than annotation
- Applications needing to identify erroneous reasoning steps
- RL training requiring step-level reward signals
When NOT to Use
- Tasks with abundant ground-truth step-level annotations
- Domains where verification is as expensive as annotation
- Simple tasks not benefiting from step-level rewards
- Outcome-only verification approaches are sufficient
Core Technique
Three-stage framework combining generation, verification, and PRM training:
# Reference-Free Process Reward Model Training
class SPARKFramework:
def __init__(self, generator, verifier):
self.generator = generator
self.verifier = verifier
self.prm = None
def stage1_generate_and_verify(self, problems, num_diverse_solutions=10):
"""
Stage 1: Generate diverse solutions and verify them.
Creates synthetic supervision without ground truth.
"""
synthetic_data = []
for problem in problems:
# Generate diverse solution trajectories
solutions = []
for _ in range(num_diverse_solutions):
solution = self.generator.generate(
problem,
temperature=0.7 # Diversity through temperature
)
solutions.append(solution)
# Parallel verification: quick pass/fail
for solution in solutions:
parallel_result = self.verifier.verify_parallel(
problem, solution
)
# Sequential verification: detailed step analysis
sequential_result = self.verifier.verify_sequential(
problem, solution
)
# Combine verification signals
verification_data = {
'problem': problem,
'solution': solution,
'steps': solution.steps,
'parallel_verdict': parallel_result,
'sequential_verdict': sequential_result,
'step_judgments': sequential_result.step_judgments
}
synthetic_data.append(verification_data)
return synthetic_data
def stage2_train_process_reward_model(self, synthetic_data):
"""
Stage 2: Train generative PRM on synthetic verification data.
Outputs are reward predictions at each step.
"""
# Initialize PRM
self.prm = GenerativePRM()
for batch in self.create_batches(synthetic_data):
problems = [d['problem'] for d in batch]
solutions = [d['solution'] for d in batch]
step_judgments = [d['step_judgments'] for d in batch]
# Forward pass: predict step-level rewards
step_predictions = self.prm.predict_step_rewards(
problems,
solutions
)
# Loss based on synthetic supervision
loss = self.compute_pairwise_loss(
step_predictions,
step_judgments
)
# Update PRM
self.prm.train_step(loss)
return self.prm
def stage3_apply_as_reward_signal(self, problems_to_solve):
"""
Stage 3: Use trained PRM as reward signal during RL.
Includes format constraints to prevent gaming.
"""
rl_agent = RLAgent()
for problem in problems_to_solve:
# Generate with RL
trajectory = rl_agent.generate_with_rl(problem)
# Compute step-level rewards using PRM
step_rewards = []
for i, step in enumerate(trajectory.steps):
# PRM predicts correctness probability for step
reward = self.prm.predict_reward(
problem,
trajectory.steps[:i+1]
)
# Add chain-of-thought verification
# Prevents output gaming by checking reasoning
verified = self.verify_step_reasoning(
problem,
trajectory.steps[:i+1]
)
# Format constraints ensure valid structure
if not self.check_format_constraints(step):
reward = 0.0 # Penalize malformed steps
step_rewards.append(reward)
# Use step rewards to train RL agent
rl_agent.update(trajectory, step_rewards)
return rl_agent
def compute_pairwise_loss(self, predictions, labels):
"""
Loss comparing predicted vs verified correctness.
Handles multiple verification signals per step.
"""
loss = 0.0
for pred, label in zip(predictions, labels):
# Log-likelihood of correct classification
loss += torch.nn.functional.binary_cross_entropy(
pred, label.float()
)
return loss / len(predictions)
def verify_step_reasoning(self, problem, steps_so_far):
"""
Secondary verification ensuring steps are logically sound.
Prevents reward hacking through malformed proofs.
"""
# Apply symbolic reasoning checks
logical_valid = self.check_logical_soundness(steps_so_far)
# Check mathematical consistency
mathematically_sound = self.check_mathematical_validity(
problem, steps_so_far
)
return logical_valid and mathematically_sound
Key insight: synthetic verification data outperforms ground-truth when used to train generative PRMs, eliminating expensive annotation requirements.
Key Results
- 67.5 F1 on ProcessBench (exceeds ground-truth: 66.4, GPT-4o: 61.9)
- 47.4% average accuracy on six mathematical reasoning benchmarks
- Reference-free training eliminates annotation bottleneck
- Works effectively with Qwen2.5-Math-7B baseline
Implementation Notes
- Generator and verifier create diverse training data
- Parallel and sequential verification provide complementary signals
- Generative PRM outputs step-level rewards
- Format constraints in RL prevent output gaming
- Chain-of-thought verification ensures reasoning validity
References
- Original paper: https://arxiv.org/abs/2512.03244
- Focus: Reference-free process reward model training
- Domain: Mathematical reasoning, reinforcement learning