# Frugal Reasoning Short Math Solutions > Achieve emergent brevity in reasoning by retaining and up-weighting easy problems during RL training, implicitly regularizing solution length without explicit penalties while maintaining accuracy on hard problems. - Skill: `adu2021/frugal-reasoning-short-math-solutions` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/frugal-reasoning-short-math-solutions` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/frugal-reasoning-short-math-solutions/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/frugal-reasoning-short-math-solutions --- # Title: Train Efficient Reasoners by Including Easy Problems During RL Standard RLVR filters out easy problems to focus on challenging ones. This creates a problem: models learn that harder problems require longer solutions, so they generate unnecessarily verbose reasoning even on solvable tasks. Frugal reasoning reverses this: by retaining moderately easy problems and up-weighting them, the model learns compact solution patterns. The result is "emergent brevity"—solutions nearly twice as short without explicit length penalties, while maintaining baseline accuracy. The mechanism is implicit length regularization through training distribution bias. ## Core Concept **Implicit Length Regularization via Easy Sample Up-Weighting**: - **Data Curriculum**: Include easy, medium, and hard problems (don't filter easy) - **Weighting Strategy**: Up-weight easy/medium problems in RL loss - **Emergent Brevity**: Model learns efficiency naturally without explicit length penalties - **Two-Stage Training**: Stage 1 learns brevity, Stage 2 improves on harder problems - **Token Budget**: Maintain fixed context limit (16K tokens typical) ## Architecture Overview - **Problem Difficulty Estimation**: Automatic or heuristic classification - **Curriculum Weighting**: Exponential down-weighting of hard problems - **GRPO Optimization**: Group Relative Policy Optimization on mixed-difficulty data - **Token Budget Constraint**: Enforce maximum sequence length - **Evaluation**: Accuracy on hard problems + length efficiency ## Implementation Steps **1. Classify Problem Difficulty** Estimate which problems are easy vs. hard. ```python class ProblemDifficultyClassifier: def __init__(self): # Simple heuristics for difficulty estimation self.difficulty_features = { 'num_digits': lambda x: x.count(str(i)) for i in range(10), 'operators': lambda x: sum(x.count(op) for op in ['+', '-', '*', '/', '%']), 'parentheses': lambda x: x.count('('), 'text_length': lambda x: len(x.split()) } def estimate_difficulty(self, problem_text, solution_length=None): """Estimate if problem is easy or hard""" features = { 'text_len': len(problem_text.split()), 'num_operators': sum(problem_text.count(op) for op in ['+', '-', '*', '/', '%']), 'has_fractions': '/' in problem_text, 'has_equations': '=' in problem_text } # Simple scoring score = ( features['text_len'] / 20 + features['num_operators'] / 3 + features['has_fractions'] * 2 + features['has_equations'] * 1 ) # Normalize to [0, 1] difficulty = min(1.0, score / 10) # Adjust by solution length if available if solution_length: # Long solutions suggest hard problems difficulty += min(0.5, solution_length / 1000) return difficulty def categorize_problems(self, problems): """Split into easy/medium/hard""" difficulties = [self.estimate_difficulty(p) for p in problems] easy_threshold = np.percentile(difficulties, 33) hard_threshold = np.percentile(difficulties, 67) categorized = { 'easy': [p for p, d in zip(problems, difficulties) if d < easy_threshold], 'medium': [p for p, d in zip(problems, difficulties) if easy_threshold <= d < hard_threshold], 'hard': [p for p, d in zip(problems, difficulties) if d >= hard_threshold] } return categorized ``` **2. Implement Curriculum Weighting** Up-weight easy/medium problems in RL loss. ```python def compute_curriculum_weights(difficulty_scores, up_weighting=2.0): """Compute importance weights for curriculum""" # Invert: easy (low difficulty) get higher weight base_weights = 1.0 - np.array(difficulty_scores) # Up-weight easy problems curriculum_weights = np.where( base_weights > 0.5, # Easy if difficulty < 0.5 base_weights * up_weighting, # Up-weight easy base_weights # Normal weight for hard ) # Normalize curriculum_weights /= curriculum_weights.mean() return curriculum_weights def compute_grpo_loss_with_curriculum( model_outputs, reference_outputs, rewards, difficulty_scores, curriculum_upweight=2.0 ): """GRPO loss weighted by curriculum""" # Compute curriculum weights weights = compute_curriculum_weights(difficulty_scores, curriculum_upweight) # Group by reward quartiles sorted_indices = np.argsort(rewards) group_size = len(rewards) // 4 loss = 0 for quartile in range(4): group_start = quartile * group_size group_end = (quartile + 1) * group_size group_indices = sorted_indices[group_start:group_end] group_rewards = torch.tensor([rewards[i] for i in group_indices]) group_weights = torch.tensor([weights[i] for i in group_indices]) # Higher quartiles are positive examples if quartile >= 2: # Positive group: maximize likelihood group_loss = -torch.log(torch.tensor([model_outputs[i] for i in group_indices]) + 1e-8) else: # Negative group: minimize likelihood group_loss = torch.log(torch.tensor([reference_outputs[i] for i in group_indices]) + 1e-8) # Weight by curriculum weighted_loss = (group_loss * group_weights).mean() loss += weighted_loss return loss / 4 ``` **3. Implement Two-Stage Training** Stage 1: Learn brevity via easy samples. Stage 2: Improve on hard problems. ```python def two_stage_frugal_training(model, dataset, token_budget=16000): """Train with frugal curriculum""" classifier = ProblemDifficultyClassifier() # Categorize problems categorized = classifier.categorize_problems([d['problem'] for d in dataset]) # Stage 1: Learn brevity with heavy easy-sample weighting print("Stage 1: Learning brevity via easy samples...") optimizer = torch.optim.Adam(model.parameters(), lr=1e-4) stage1_data = categorized['easy'] + categorized['medium'] # Mix easy and medium stage1_weights = compute_curriculum_weights( [classifier.estimate_difficulty(p) for p in stage1_data], up_weighting=3.0 # Heavy up-weighting for brevity ) for epoch in range(3): for problem, weight in zip(stage1_data, stage1_weights): # Generate solution with token budget solution = model.generate(problem, max_tokens=token_budget) # Reward: correctness + brevity correct = verify_solution(solution, problem) length_efficiency = 1.0 / (1.0 + len(solution.split()) / 100) reward = 0.8 * correct + 0.2 * length_efficiency # Loss weighted by curriculum loss = -torch.log(torch.tensor(reward)) * weight optimizer.zero_grad() loss.backward() optimizer.step() # Stage 2: Improve on hard problems with learned brevity print("Stage 2: Improving on hard problems...") stage2_data = categorized['hard'] stage2_weights = compute_curriculum_weights( [classifier.estimate_difficulty(p) for p in stage2_data], up_weighting=1.0 # Normal weighting for hard problems ) for epoch in range(10): for problem, weight in zip(stage2_data, stage2_weights): # Generate with token budget (model has learned brevity) solution = model.generate(problem, max_tokens=token_budget) # Reward: emphasis on correctness correct = verify_solution(solution, problem) length_efficiency = 1.0 / (1.0 + len(solution.split()) / 100) reward = 0.95 * correct + 0.05 * length_efficiency loss = -torch.log(torch.tensor(reward)) * weight optimizer.zero_grad() loss.backward() optimizer.step() return model ``` **4. Evaluate Emergent Brevity** Measure length and accuracy trade-off. ```python def evaluate_frugal_reasoning(model, test_problems, ground_truth_answers): """Evaluate model on length and accuracy""" results = { 'accuracies': [], 'solution_lengths': [], 'tokens_per_problem': [] } for problem, gt_answer in zip(test_problems, ground_truth_answers): # Generate solution solution = model.generate(problem, max_tokens=16000) # Check correctness predicted_answer = extract_answer(solution) correct = predicted_answer == gt_answer results['accuracies'].append(correct) results['solution_lengths'].append(len(solution.split())) results['tokens_per_problem'].append(len(solution.split()) * 1.3) # Token estimate # Compute metrics accuracy = np.mean(results['accuracies']) avg_length = np.mean(results['solution_lengths']) efficiency = np.mean(results['tokens_per_problem']) print(f"Accuracy: {accuracy:.1%}") print(f"Avg solution length: {avg_length:.0f} words") print(f"Token efficiency: {efficiency:.0f} tokens/problem") return results ``` ## Practical Guidance **When to Use**: - Mathematical reasoning where solution length correlates with problem hardness - Settings with token budgets or latency constraints - Training reasoning models post-SFT **Hyperparameters**: - curriculum_upweight: 2-3 (how much to favor easy samples) - token_budget: 16K typical for math, adjust for domain - easy_problem_ratio: 0.4-0.6 of training set **When NOT to Use**: - Domains where complexity doesn't map to problem difficulty - Models without clear solution length patterns - Settings requiring verbose reasoning for auditing **Pitfalls**: - **Oversimplification**: Model learns to give short wrong answers; mitigate with correctness rewards - **Difficulty estimation failure**: Heuristic difficulty estimates may be inaccurate; validate - **Stage 1 undertraining**: If stage 1 doesn't converge to brevity, stage 2 won't benefit **Key Insight**: The mechanism works because easy samples teach "fast solution patterns." Once the model learns to solve easy problems quickly, these patterns transfer to hard problems. No explicit length penalty needed. ## Reference arXiv: https://arxiv.org/abs/2511.01937