# Opensir Self Play Reasoning > Enable open-ended mathematical learning through co-evolutionary teacher-student self-play where a single model alternates roles to generate progressively harder problems and solve them, creating a virtuous cycle without external data. - Skill: `adu2021/opensir-self-play-reasoning` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/opensir-self-play-reasoning` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/opensir-self-play-reasoning/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/opensir-self-play-reasoning --- # Title: Bootstrap Mathematical Reasoning From Seed Problems Without Annotation Traditional supervised learning requires massive annotated datasets. OpenSIR achieves autonomous improvement through **co-evolutionary self-play**: a single model acts as both problem-setter (teacher) and problem-solver (student), generating novel problems calibrated to current capability and solving them. The framework requires only a small seed of initial problems, then bootstraps to arbitrary complexity. The key is maintaining optimal difficulty: problems easy enough to learn from but hard enough to improve capability. ## Core Concept **Co-Evolutionary Teacher-Student Self-Play**: - **Single Policy, Two Roles**: Model alternates between generating problems (teacher) and solving them (student) - **Difficulty Calibration**: Teacher learns to generate problems at ~70% solve rate (optimal difficulty) - **Diversity Rewards**: Encourage distinct mathematical concepts vs. repeating same patterns - **No External Supervision**: Learn entirely from self-generated data - **Emergent Curriculum**: Naturally progress from arithmetic to advanced mathematics ## Architecture Overview - **Problem Generator (Teacher)**: Produces novel math problems conditioned on difficulty level and concept diversity - **Problem Solver (Student)**: Generates step-by-step solutions via reasoning - **Capability Estimator**: Tracks model performance across problem categories - **Curriculum Controller**: Adjusts teacher difficulty based on student performance - **Scoring System**: Rewards for both solution accuracy and problem novelty ## Implementation Steps **1. Implement Teacher (Problem Generation)** Design the teacher to generate diverse, well-formed mathematical problems. ```python class MathTeacher(nn.Module): def __init__(self, model, difficulty_levels=[1, 2, 3, 4, 5]): self.model = model self.difficulty_levels = difficulty_levels self.concept_tracker = defaultdict(int) def generate_problem(self, concepts=None, difficulty=None): if difficulty is None: difficulty = np.random.choice(self.difficulty_levels) # Prompt teacher to generate problem prompt = f"""Generate a {difficulty}/5 difficulty math problem. Previous problems covered: {list(self.concept_tracker.keys())} Please avoid repeating the same concepts repeatedly. Problem:""" problem = self.model.generate(prompt, max_tokens=200) # Track concepts used if concepts: self.concept_tracker[concepts] += 1 return problem, difficulty def sample_problems(self, num_problems=10): # Generate diverse problems across difficulties problems = [] for _ in range(num_problems): # Vary difficulty difficulty = np.random.choice(self.difficulty_levels) problem, _ = self.generate_problem(difficulty=difficulty) problems.append(problem) return problems ``` **2. Implement Student (Problem Solver)** Create the solver that learns from generated problems. ```python class MathStudent(nn.Module): def __init__(self, model): self.model = model self.performance_tracker = defaultdict(lambda: {'solved': 0, 'attempts': 0}) def solve_problem(self, problem, return_confidence=False): # Prompt model to solve problem with step-by-step reasoning prompt = f"""Solve this problem step-by-step: {problem} Solution:""" solution = self.model.generate(prompt, max_tokens=500) # Extract final answer answer = self.extract_answer(solution) if return_confidence: # Get confidence score from model confidence_prompt = f"How confident are you in this answer (0-1)? {answer}" confidence = float(self.model.generate(confidence_prompt, max_tokens=10)) return solution, answer, confidence return solution, answer def extract_answer(self, solution): # Parse answer from solution text (problem-dependent) # Try multiple extraction strategies if 'answer is' in solution: return solution.split('answer is')[-1].strip() if '=' in solution: return solution.split('=')[-1].strip() return solution.split()[-1] def verify_answer(self, problem, answer): # Use ground truth verification or self-consistency # For now, simplified prompt = f"Is '{answer}' the correct answer to '{problem}'? Yes/No:" response = self.model.generate(prompt, max_tokens=10) return 'yes' in response.lower() def track_performance(self, category, solved): self.performance_tracker[category]['attempts'] += 1 if solved: self.performance_tracker[category]['solved'] += 1 ``` **3. Implement Co-Evolution Training Loop** Train both teacher and student through self-play. ```python def co-evolutionary_training(model, initial_problems, num_iterations=100): teacher = MathTeacher(model) student = MathStudent(model) # Initialize from seed problems problem_pool = initial_problems.copy() for iteration in range(num_iterations): # Teacher phase: generate new problems new_problems = teacher.sample_problems(num_problems=5) problem_pool.extend(new_problems) # Student phase: solve problems from pool solve_correct = 0 solve_total = 0 for problem in problem_pool[-50:]: # Recent problems solution, answer, confidence = student.solve_problem( problem, return_confidence=True ) verified = student.verify_answer(problem, answer) solve_total += 1 if verified: solve_correct += 1 # Track performance category = extract_category(problem) # algebra, geometry, etc. student.track_performance(category, verified) # Compute solve rate solve_rate = solve_correct / solve_total if solve_total > 0 else 0 # Curriculum adjustment # If solve_rate > 0.7, increase difficulty # If solve_rate < 0.5, decrease difficulty if solve_rate > 0.7: teacher.difficulty_levels = [d + 0.1 for d in teacher.difficulty_levels] elif solve_rate < 0.5: teacher.difficulty_levels = [max(1, d - 0.1) for d in teacher.difficulty_levels] print(f"Iteration {iteration}: Solve Rate {solve_rate:.2%}, Pool Size {len(problem_pool)}") # Periodic reinforcement signal if iteration % 10 == 0: # Train on successful problem-solution pairs successful_pairs = [ (p, s) for p, s in problem_solution_pairs if verified[p] ] # Fine-tune on successful patterns update_model(model, successful_pairs) ``` **4. Implement Diversity and Difficulty Rewards** Ensure teacher generates diverse problems while maintaining difficulty calibration. ```python def compute_teacher_reward(new_problem, problem_pool, solve_rate, category_coverage): # Reward 1: Difficulty calibration # Want solve rate around 70% difficulty_reward = 1.0 - abs(solve_rate - 0.7) # Reward 2: Diversity # Penalize similar problems similarity_to_existing = max( cosine_similarity(embed(new_problem), embed(p)) for p in problem_pool[-20:] ) diversity_reward = 1.0 - 0.5 * similarity_to_existing # Reward 3: Coverage # Encourage unexplored categories new_category = extract_category(new_problem) coverage_reward = 1.0 / (1.0 + category_coverage[new_category]) total_reward = 0.6 * difficulty_reward + 0.3 * diversity_reward + 0.1 * coverage_reward return total_reward ``` ## Practical Guidance **When to Use**: - Mathematical reasoning tasks - Domains where problem generation is straightforward - Building reasoning capabilities from minimal seed data - Educational applications requiring progressive difficulty **Hyperparameters**: - target_solve_rate: 0.7 (70% correctness for optimal difficulty) - difficulty_adjustment_step: 0.1 per iteration - problem_pool_retention: Keep last 50-100 problems for training **When NOT to Use**: - Domains where problem generation requires human judgment (e.g., physics) - Tasks with complex evaluation metrics - Real-time applications (self-play generation is slow) **Pitfalls**: - **Mode collapse**: Teacher converges to generating same problem repeatedly; use diversity rewards - **Easy problems dominating**: Student solves simple problems but skill doesn't transfer; enforce difficulty growth - **Verification errors**: If answer verification is wrong, learning goes off-track; use multiple verification strategies **Key Insight**: Co-evolution only works if teacher and student improve together. If one diverges, the loop breaks. Monitor their relative performance. ## Reference arXiv: https://arxiv.org/abs/2511.00602