Swe Lego Issue Resolution

Overview

SWE-Lego demonstrates that sophisticated software engineering agent capabilities can be achieved through a carefully designed lightweight approach: supervised fine-tuning (SFT) only, without requiring complex multi-stage training pipelines (mid-training, SFT, RL combinations).

Core Insight: State-of-the-art software issue resolution doesn't require elaborate training paradigms. Instead, three key components—high-quality data, refined SFT procedure, and test-time scaling—deliver competitive performance at fraction of typical complexity.

Three Building Blocks

Block 1: SWE-Lego Dataset (32k + 18k trajectories)

A carefully curated dataset combining real and synthetic data for both quality and quantity:

Composition:

• 32,000 task instances - High-quality, diverse software engineering tasks
• 18,000 validated trajectories - Correct solution paths for training
• Real data - Authentic GitHub issues and PRs
• Synthetic data - Complementary examples filling capability gaps

Quality Control:

• Trajectory validation ensuring correctness
• Diversity across programming languages and issue types
• Difficulty stratification for curriculum learning
• Removal of low-quality or ambiguous examples

Key Advantage: Balanced coverage of both common patterns and edge cases through synthetic-real combination.

Block 2: Refined SFT Procedure

Enhanced training methodology improving action quality and overall performance:

Error Masking: Filter trajectories containing invalid tool calls or impossible actions. Prevents model learning degenerate patterns.

def mask_errors_in_trajectory(trajectory):
    """Filter out invalid actions from training trajectory."""
    valid_actions = []
    for step in trajectory:
        if is_valid_action(step.action, step.context):
            valid_actions.append(step)
        else:
            # Skip invalid action; continue from next valid state
            pass
    return valid_actions

Difficulty-Based Curriculum: Train on progressively harder examples to improve convergence and final performance:

# Stage 1: Simple fixes (syntax errors, type issues)
curriculum_stage_1 = [
    t for t in dataset
    if task_difficulty(t.issue) <= "easy"
]

# Stage 2: Medium complexity (logic bugs, simple refactoring)
curriculum_stage_2 = [
    t for t in dataset
    if "easy" < task_difficulty(t.issue) <= "medium"
]

# Stage 3: Complex (architectural changes, deep reasoning)
curriculum_stage_3 = [
    t for t in dataset
    if task_difficulty(t.issue) > "medium"
]

train_sft(curriculum_stage_1)  # Epochs 1-2
train_sft(curriculum_stage_2)  # Epochs 3-4
train_sft(curriculum_stage_3)  # Epochs 5+

Results from Blocks 1+2:

• SWE-Lego-Qwen3-8B: 42.2% on SWE-Bench Verified
• SWE-Lego-Qwen3-32B: 52.6% on SWE-Bench Verified
• State-of-the-art among open-source models

Block 3: Test-Time Scaling

Improve performance at inference through verification and multiple attempts:

Verification-Based Scaling:

• Generate multiple solution candidates
• Use trained verifier to assess solution quality
• Select highest-confidence solution

def test_time_scale(problem, verifier, num_attempts=16):
    """Generate multiple solutions and select best."""
    candidates = []

    for _ in range(num_attempts):
        solution = model.generate_solution(problem)
        confidence = verifier.score(problem, solution)
        candidates.append((solution, confidence))

    # Return highest-confidence solution
    best_solution, _ = max(candidates, key=lambda x: x[1])
    return best_solution

Performance Gains with Test-Time Scaling:

• SWE-Lego-Qwen3-8B: 42.2% → 49.6% (TTS@16)
• SWE-Lego-Qwen3-32B: 52.6% → 58.8% (TTS@16)
• Linear scaling up to 16 attempts; diminishing returns beyond

Computational Trade-off:

• 16x increase in inference compute for 7-9% accuracy gain
• Practical for offline analysis; less suitable for real-time deployment

Benchmark Performance

SWE-Bench Verified (rigorous subset with validated fixes):

• SWE-Lego-Qwen3-8B: 42.2% baseline, 49.6% with TTS@16
• SWE-Lego-Qwen3-32B: 52.6% baseline, 58.8% with TTS@16
• Competitive with much larger closed-source models

Comparison to Complex Pipelines:

• ReTool (SFT + RL): $10,000 training cost
• SWE-Lego (SFT only): <$1,000 training cost
• SWE-Lego achieves comparable or superior performance

When to Use SWE-Lego Approach

Use when:

• Fine-tuning models for software engineering tasks
• Building issue-resolution agents with limited compute budgets
• Need interpretable training pipeline (SFT only, no RL complexity)
• Working with open-source models (Qwen, Llama, etc.)
• Test-time compute is available (multiple solution generation acceptable)

When NOT to use:

• Real-time issue resolution requiring single-pass inference
• Extremely limited fine-tuning data (<1,000 examples)
• Scenarios where interpretability of training process isn't important
• Proprietary models where fine-tuning isn't accessible

Implementation Lessons

Dataset Design:

• Mix real and synthetic data strategically
• Validate trajectories before inclusion
• Stratify by difficulty for curriculum learning
• Remove ambiguous or contradictory examples

Training Procedure:

• Filter invalid actions from training trajectories
• Use curriculum learning for complex tasks
• Monitor both intermediate action quality and final correctness
• Iterate on data quality more than model size

Inference Strategy:

• Trained verifier crucial for test-time scaling effectiveness
• 16 attempts provides good accuracy/compute tradeoff
• Diminishing returns beyond; batch processing is efficient

Related Approaches

SFT-Only Paradigm:

• Minimal training complexity vs. SFT + RL combinations
• Easier to reproduce and debug
• More interpretable training dynamics

Multi-Stage Training:

• More sophisticated but harder to optimize
• Higher computational cost
• Potential for instability (entropy explosion, etc.)

Code Availability

Project website: https://github.com/SWE-Lego/swe-lego

Public Resources:

• SWE-Lego dataset (32k instances, 18k trajectories)
• Fine-tuned models (8B and 32B variants)
• Training scripts and curriculum implementation
• Verifier models for test-time scaling

References

• SWE-Lego-Qwen3-32B: 52.6% on SWE-Bench Verified (open-source SOTA)
• Test-time scaling reaches 58.8% (TTS@16)
• Lightweight SFT-only approach outperforms complex multi-stage pipelines
• 10x lower training cost than competing methods