# Auto Codebench Generator > Automatically generates diverse multilingual code benchmarks using LLMs, creating 3920 problems across 20 programming languages with quality assurance filtering. - Skill: `adu2021/auto-codebench-generator` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/auto-codebench-generator` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/auto-codebench-generator/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/auto-codebench-generator --- ## AutoCodeBench: LLMs as Code Benchmark Generators ### Core Concept AutoCodeBench enables automatic generation of code benchmark datasets without manual annotation. The approach leverages LLMs to generate problem statements, creates test cases autonomously, and applies quality filtering to ensure correctness. This creates diverse, multilingual benchmarks addressing limitations in existing datasets that focus primarily on Python. ### Architecture Overview - **LLM-Based Problem Generation**: Generate diverse coding problems across domains - **Autonomous Test Case Creation**: Generate inputs and execute code to obtain outputs - **Multilingual Coverage**: Support 20+ programming languages - **Quality Assurance Pipeline**: Reverse-order generation and filtering steps - **Difficulty Scaling**: Problems from easy to hard ### Implementation Steps **Step 1: Implement Problem Generation** Generate coding problems automatically: ```python class ProblemGenerator: def __init__(self, llm_model): super().__init__() self.llm = llm_model self.difficulty_levels = ['easy', 'medium', 'hard'] self.categories = ['strings', 'arrays', 'graphs', 'math', 'dp'] def generate_problem(self, category, difficulty): """Generate single problem with statement and examples.""" prompt = f"""Generate a {difficulty} {category} programming problem. Include: 1. Clear problem statement 2. Constraints 3. Example input/output Return JSON with 'statement' and 'examples' keys.""" with torch.no_grad(): response = self.llm.generate(prompt, max_length=500) return self._parse_problem_response(response) def _parse_problem_response(self, response): """Parse generated problem.""" import json try: return json.loads(response) except: return {'statement': response, 'examples': []} def generate_problems_batch(self, num_problems, languages=20): """Generate batch of problems.""" problems = [] for i in range(num_problems): category = self.categories[i % len(self.categories)] difficulty = self.difficulty_levels[i % len(self.difficulty_levels)] problem = self.generate_problem(category, difficulty) problems.append(problem) return problems ``` **Step 2: Implement Test Case Generation** Create test inputs and outputs: ```python class TestCaseGenerator: def __init__(self, sandbox_executor): super().__init__() self.executor = sandbox_executor def generate_test_cases(self, problem_statement, solution_code, num_tests=5): """Generate test cases by executing solution.""" test_cases = [] # Generate diverse inputs inputs = self._generate_diverse_inputs(problem_statement, num_tests) for test_input in inputs: try: # Execute solution in sandbox output = self.executor.execute(solution_code, test_input) test_cases.append({ 'input': test_input, 'output': output, 'valid': True }) except Exception as e: test_cases.append({ 'input': test_input, 'error': str(e), 'valid': False }) return [t for t in test_cases if t['valid']] def _generate_diverse_inputs(self, problem_stmt, num): """Generate diverse test inputs.""" inputs = [] # Edge cases inputs.append(self._generate_edge_case(problem_stmt)) # Random inputs for _ in range(num - 1): inputs.append(self._generate_random_input(problem_stmt)) return inputs def _generate_edge_case(self, stmt): """Generate edge case input.""" if 'empty' in stmt.lower(): return '[]' elif 'single' in stmt.lower(): return '[1]' else: return '[]' def _generate_random_input(self, stmt): """Generate random input.""" import random size = random.randint(1, 100) values = [random.randint(0, 100) for _ in range(size)] return str(values) ``` **Step 3: Implement Quality Filtering** Ensure benchmark quality: ```python class BenchmarkQualityFilter: def __init__(self): super().__init__() def filter_problems(self, problems, test_cases): """Filter out low-quality problems.""" filtered = [] for problem, tests in zip(problems, test_cases): if self._is_high_quality(problem, tests): filtered.append((problem, tests)) return filtered def _is_high_quality(self, problem, tests): """Assess problem quality.""" # Check 1: Problem clarity clarity = len(problem.get('statement', '').split()) > 20 if not clarity: return False # Check 2: Valid test cases valid_tests = len([t for t in tests if t.get('valid')]) if valid_tests < 2: return False # Check 3: Not too easy/hard complexity = self._estimate_complexity(problem) if complexity < 1 or complexity > 10: return False return True def _estimate_complexity(self, problem): """Estimate problem complexity.""" stmt = problem.get('statement', '') complexity_indicators = ['loop', 'recursion', 'graph', 'dynamic'] return sum(1 for ind in complexity_indicators if ind in stmt.lower()) def apply_reverse_verification(self, problem, tests): """Verify problem by generating solution and testing.""" # Generate solution attempt # Verify solution passes tests return True ``` **Step 4: Build Final Benchmark Dataset** Assemble completed benchmark: ```python class BenchmarkAssembler: def assemble_benchmark(self, filtered_problems, languages=20): """Create final benchmark across languages.""" benchmark = { 'problems': [], 'statistics': {}, 'language_distribution': {} } for problem, tests in filtered_problems: benchmark_problem = { 'problem_id': str(uuid.uuid4()), 'statement': problem['statement'], 'test_cases': tests, 'languages': languages, 'difficulty': self._assess_difficulty(problem) } benchmark['problems'].append(benchmark_problem) # Compute statistics benchmark['statistics'] = { 'total_problems': len(benchmark['problems']), 'total_languages': languages, 'total_test_cases': sum(len(p['test_cases']) for p in benchmark['problems']) } return benchmark ``` ### Practical Guidance **Hyperparameters and Configuration**: - Number of problems: 3920 (20 languages × 196 problems) - Test cases per problem: 3-10 - Language coverage: 20+ languages - Difficulty distribution: 30% easy, 50% medium, 20% hard **When to Use AutoCodeBench**: - Creating diverse multilingual code evaluation benchmarks - Evaluating LLMs across multiple languages - Addressing limitations of Python-only benchmarks - Building domain-specific code datasets **When NOT to Use**: - Specialized domain benchmarks requiring domain expertise - When human expert validation is critical - Niche languages with limited LLM support **Implementation Notes**: - Quality filtering critical for benchmark usefulness - Sandbox execution prevents security issues - Reverse verification ensures correctness - Language diversity essential for comprehensive evaluation ### Reference Paper: AutoCodeBench: LLMs as Code Benchmark Generators ArXiv: 2508.09101 Performance: Created 3920 problems across 20 languages; even advanced models struggle with complexity and multilingual nature