Opencodeinstruct Eval

opencodeinstruct-eval

OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs — Ahmad et al. (2025) (arXiv:2504.04030, 2025)

What this evaluates

Evaluates the code generation, algorithmic problem-solving, and complex function-calling capabilities of instruction-tuned LLMs across multiple standardized coding benchmarks.

Datasets

• HumanEval — total ?; splits: test (-1)
• MBPP — total ?; splits: test (-1)
• LiveCodeBench — total 713; splits: test (713)
• BigIntCodeBench-Instruct — total 1140; splits: test (1140)

Metrics

• pass@1 (primary) — range: percent
  • Percentage of generated solutions that pass all provided unit tests for a given problem. Computed as the fraction of problems where at least one generated sample passes all test cases.

Input / output format

Input: Natural language problem description, often including a function signature or stub, and sometimes library constraints or competitive programming context.

Output: Python code implementing the requested function or script.

Scoring recipe

def compute_pass_at_1(predictions, gold_tests):
    passed_count = 0
    for code, tests in zip(predictions, gold_tests):
        if execute_and_check(code, tests):
            passed_count += 1
    return (passed_count / len(predictions)) * 100

Common pitfalls

• Continuously updated benchmarks like LiveCodeBench require strict contamination checks to ensure test cases were not in the training data.
• pass@1 scores can be inflated by weak or incomplete unit tests; some benchmarks use minimal test coverage.

Evidence (verbatim from paper)

HumanEval and MBPP We reported the evaluations on HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), HumaneEval+ (Liu et al., 2023), and MBPP+ (Liu et al., 2023) which are the most common benchmarks for function-level code generation. LiveCodeBench (Jain et al., 2025) is an extensive, contamination-free benchmark created to assess the coding capabilities of LLMs. BigIntCodeBench-Instruct, a natural language instruction adaptation of BigCodeBench (Zhuo et al., 2025), challenges LLMs with complex function calling tasks.

Citation

@misc{ahmad2025opencodeinstruct,
  title={OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs},
  author={Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2504.04030}
}

• arXiv: 2504.04030