adu2021 SWE-Factory: Automated Benchmark Generation
Core Concept
SWE-Factory fully automates GitHub issue resolution dataset construction through a multi-agent LLM system that coordinates environment setup, binary file recovery, and fail2pass validation. Instead of manual inspection and custom log parsing, the system uses exit-code-based testing (standardized Unix return codes), successfully generating 337 valid benchmarks from 671 issues across Python, Java, JavaScript, and TypeScript at $0.047 per instance and 96% quality.
Architecture Overview
- SWE-Builder Multi-Agent System: Four collaborative agents (Repository Explorer, Environment Manager, Test Manager, Test Analyst) with shared Environment Memory Pool for configuration reuse
- Binary File Recovery: Automatically downloads missing test binaries and removes incomplete hunks from patches
- Exit-Code-Based Validation: Standardized fail2pass testing via Unix exit codes ($rc) instead of manual log parsing, achieving F1=0.99
- Environment Memory Pool: Caches successful Docker configurations from similar repository versions, reducing redundant iterations
- Language Support: Python, Java, JavaScript, TypeScript with unified testing interface
Implementation
Step 1: Binary File Recovery System
import os
import subprocess
from pathlib import Path
class BinaryFileRecoverer:
"""
Automatically downloads missing binary test files from repositories
and removes incomplete binary hunks from patches.
"""
def __init__(self, repo_path, repo_url):
self.repo_path = repo_path
self.repo_url = repo_url
def identify_missing_binaries(self, patch_content):
"""
Parse patch file to identify missing binary files.
Binary patches show "Binary files ... differ" marker.
"""
missing_binaries = []
lines = patch_content.split('\n')
for i, line in enumerate(lines):
if 'Binary files' in line and 'differ' in line:
# Extract filename from patch header
for j in range(i-1, max(0, i-10), -1):
if lines[j].startswith('+++'):
filename = lines[j].split('\t')[0].replace('+++ b/', '')
missing_binaries.append(filename)
break
return missing_binaries
def download_binaries_from_repo(self, filenames):
"""
Download binary files from original repository.
Handles different VCS (GitHub, GitLab, etc.)
"""
downloaded = []
for filename in filenames:
# Construct raw file URL
raw_url = self.repo_url.replace('github.com', 'raw.githubusercontent.com')
raw_url = f"{raw_url}/HEAD/{filename}"
try:
result = subprocess.run(
['curl', '-L', '-o', filename, raw_url],
cwd=self.repo_path,
timeout=30,
capture_output=True
)
if result.returncode == 0 and os.path.exists(os.path.join(self.repo_path, filename)):
downloaded.append(filename)
print(f"Downloaded: {filename}")
except Exception as e:
print(f"Failed to download {filename}: {e}")
return downloaded
def clean_patch_binaries(self, patch_content):
"""
Remove incomplete binary hunks from patches.
Keeps only well-formed binary sections.
"""
lines = patch_content.split('\n')
cleaned = []
skip_binary_section = False
for i, line in enumerate(lines):
if 'Binary files' in line and 'differ' in line:
skip_binary_section = True
cleaned.append(line) # Keep the "differs" marker
elif line.startswith('+++') or line.startswith('---'):
skip_binary_section = False
cleaned.append(line)
elif not skip_binary_section or line.startswith(('GIT binary patch', 'literal')):
cleaned.append(line)
return '\n'.join(cleaned)
Step 2: Multi-Agent Environment Builder
class SWEBuilderMultiAgent:
"""
Orchestrates four collaborative agents to construct evaluation environment.
Uses shared memory pool for configuration reuse.
"""
def __init__(self, repo_path, model_name='gpt-4-mini'):
self.repo_path = repo_path
self.model = model_name
self.env_memory_pool = {}
def repository_explorer(self):
"""Agent 1: Collects environment dependencies and test commands."""
exploration_prompt = f"""
Analyze repository at {self.repo_path}:
1. Identify all dependency files (requirements.txt, pom.xml, package.json, etc.)
2. Extract all test commands (pytest, maven, npm test, etc.)
3. List any configuration files (setup.cfg, tsconfig.json, etc.)
4. Identify language versions needed (Python 3.9+, Java 11+, etc.)
Return structured JSON with sections: dependencies, test_commands, configs, versions
"""
return self._llm_call(exploration_prompt)
def environment_manager(self, exploration_result):
"""Agent 2: Generates Dockerfile configuration."""
docker_prompt = f"""
Based on repository analysis:
{exploration_result}
Generate complete Dockerfile that:
1. Installs base image matching language version requirements
2. Installs all dependencies in correct order
3. Sets environment variables
4. Copies repository code
5. Runs any setup scripts
Output valid Dockerfile content.
"""
docker_content = self._llm_call(docker_prompt)
# Check memory pool for similar configurations
similar = self._find_similar_config(exploration_result)
if similar:
print(f"Found similar configuration: {similar}")
docker_content = self._merge_configs(docker_content, similar)
return docker_content
def test_manager(self, exploration_result):
"""Agent 3: Creates executable test scripts with exit code reporting."""
test_prompt = f"""
Based on identified test commands:
{exploration_result['test_commands']}
Create a master test script that:
1. Sets up test environment
2. Runs each test command sequentially
3. Captures exit codes
4. Appends standardized marker: OMNIGRIL_EXIT_CODE=$rc
5. Returns appropriate exit code (0=pass, 1=fail)
Language: {self._detect_language()}
"""
test_script = self._llm_call(test_prompt)
return test_script
def test_analyst(self, docker_content, test_script):
"""Agent 4: Validates environment and orchestrates refinement."""
# Try to build Docker image
build_success = self._build_docker(docker_content)
if not build_success:
# Iteratively refine via LLM
refinement_prompt = f"""
Docker build failed. Previous Dockerfile:
{docker_content}
Common issues:
- Missing base image layer
- Dependency installation order
- Working directory not set
Provide corrected Dockerfile.
"""
docker_content = self._llm_call(refinement_prompt)
return self.test_analyst(docker_content, test_script)
# Build passed; store in memory pool
self.env_memory_pool[hash(docker_content)] = {
'dockerfile': docker_content,
'test_script': test_script,
'success': True
}
return {'dockerfile': docker_content, 'test_script': test_script}
def _find_similar_config(self, exploration_result):
"""Find similar configurations in memory pool."""
repo_hash = hash(exploration_result.get('dependencies', ''))
# Simple similarity: same language/version
for stored_hash, config in self.env_memory_pool.items():
if config['success']:
return config
return None
def _merge_configs(self, new_config, similar_config):
"""Merge new config with similar successful one."""
# Simplified: prefer new, fallback to similar
return new_config
def _llm_call(self, prompt):
"""Call LLM with prompt."""
# Simplified placeholder
return f"Generated from: {prompt[:50]}..."
def _detect_language(self):
"""Detect primary language of repository."""
if os.path.exists(os.path.join(self.repo_path, 'requirements.txt')):
return 'python'
elif os.path.exists(os.path.join(self.repo_path, 'pom.xml')):
return 'java'
elif os.path.exists(os.path.join(self.repo_path, 'package.json')):
return 'javascript'
return 'unknown'
def _build_docker(self, dockerfile_content):
"""Test build Docker image."""
# Simplified: return True for now
return True
Step 3: Exit-Code-Based Fail2Pass Validation
import re
import subprocess
class ExitCodeValidator:
"""
Standardized fail2pass validation using Unix exit codes.
Eliminates custom log parsing; achieves 100% precision, 98.3% recall.
"""
def create_test_wrapper(self, original_test_command, language):
"""
Wraps test command to append standardized exit code marker.
Format: OMNIGRIL_EXIT_CODE=$rc
"""
if language == 'python':
wrapper = f"""#!/bin/bash
set -e
bash -c '{original_test_command}'
rc=$?
echo "OMNIGRIL_EXIT_CODE=$rc"
exit $rc
"""
elif language == 'java':
wrapper = f"""#!/bin/bash
set -e
{original_test_command}
rc=$?
echo "OMNIGRIL_EXIT_CODE=$rc"
exit $rc
"""
else:
wrapper = f"""#!/bin/bash
set -e
{original_test_command}
rc=$?
echo "OMNIGRIL_EXIT_CODE=$rc"
exit $rc
"""
return wrapper
def extract_exit_code(self, test_output):
"""
Parse test output and extract exit code marker.
Returns (exit_code, is_valid).
"""
pattern = r'OMNIGRIL_EXIT_CODE=(\d+)'
match = re.search(pattern, test_output)
if match:
exit_code = int(match.group(1))
return exit_code, True
return None, False
def validate_fail2pass(self, buggy_version_output, fixed_version_output):
"""
Verify that test failed on buggy version and passed on fixed version.
Returns (is_valid, buggy_code, fixed_code).
"""
buggy_code, buggy_valid = self.extract_exit_code(buggy_version_output)
fixed_code, fixed_valid = self.extract_exit_code(fixed_version_output)
if not (buggy_valid and fixed_valid):
return False, None, None
# Fail2pass: buggy version failed (non-zero), fixed version passed (zero)
is_fail2pass = (buggy_code != 0) and (fixed_code == 0)
return is_fail2pass, buggy_code, fixed_code
Practical Guidance
Dataset Construction Workflow:
- Collect GitHub issues (use GitHub API with issue labels like "bug", "fix")
- Extract patches from issue comments or linked PRs
- Run Binary Recovery to handle test fixtures
- Execute SWE-Builder multi-agent pipeline to construct environment
- Validate with Exit-Code-Based testing
Cost Optimization:
- Average: $0.047 per valid instance with GPT-4 mini
- Batch processing: Process 50+ issues in parallel to amortize LLM calls
- Memory pool: Reuse successful configurations reduces total cost by 30%
Quality Assurance:
- Validate exit codes independently with 2+ test runners
- Manual review of first 50 instances to verify quality
- Monitor F1 score (0.99 target) on exit code extraction
Supported Languages:
- Python: pytest, unittest, nose
- Java: Maven, Gradle
- JavaScript/TypeScript: Jest, Mocha, npm test
- Others: Extensible via custom test runners
Reference
- Unix exit codes: Standard convention (0=success, non-zero=failure) across all languages
- Docker layer caching: Significantly speeds up repeated builds for similar configurations
- Environment Memory Pool: Inspired by case-based reasoning; dramatically improves efficiency