Humaneval Mbpp Eval

humaneval-mbpp-eval

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement — Zheng et al. (2024) (arXiv:2402.14658, 2024)

What this evaluates

Evaluates a model's ability to generate correct Python code for programming tasks and iteratively refine it using execution feedback or simulated human guidance. It measures both initial code generation quality and the effectiveness of a multi-turn debugging loop under strict runtime and edge-case constraints.

Datasets

• HumanEval — total ?; splits: test (-1)
• MBPP — total ?; splits: test (-1)
• HumanEval+ — total ?; splits: test (-1)
• MBPP+ — total ?; splits: test (-1)

Metrics

• pass@1 (primary) — range: percent
  • The percentage of tasks for which the generated code passes all provided unit tests. Evaluated via greedy decoding for single-turn, and up to two refinement rounds for multi-turn scenarios.

Input / output format

Input: Task description/prompt. For multi-turn: previous code attempt, execution results (exceptions, expected vs actual outputs, or timeout messages), and optional synthetic human feedback.

Output: Python code snippet solving the task.

Scoring recipe

def score(predictions, gold_tests):
    passed = 0
    for pred in predictions:
        sanitized = evalplus.sanitize(pred)
        for _ in range(2):
            if run_tests(sanitized, gold_tests):
                passed += 1
                break
            feedback = get_execution_feedback(sanitized, gold_tests)
            sanitized = refine_code(sanitized, feedback)
    return (passed / len(predictions)) * 100

Common pitfalls

• Using the original HumanEval/MBPP test suites instead of EvalPlus, which significantly underestimates performance due to missing edge cases.
• Ignoring the multi-turn refinement limit (max 2 rounds) when calculating pass rates, leading to inflated scores.
• Not applying the EvalPlus unified sanitizer to generated code before execution, which can cause false negatives on syntactically valid but non-standard code.

Evidence (verbatim from paper)

For single-turn code generation, we craft a simple instruction to encapsulate the original prompt, forming a new input for the model. The exact prompts are detailed in Appendix D, and we assess the model's performance using the pass@1 metric, as per EvalPlus's guidelines.

Citation

@misc{zheng2024opencodeinterpreter,
  title={OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement},
  author={Zheng et al. (2024)},
  year={2024},
  note={arXiv:2402.14658}
}

• arXiv: 2402.14658