Kodcode Eval

kodcode-eval

KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding — Zhangchen Xu et al. (2025) (arXiv:2503.02951, 2025)

What this evaluates

Evaluates the functional correctness and robustness of code generation models on diverse programming tasks, including standard algorithmic problems, external library usage, and competitive programming challenges.

Datasets

• HumanEval(+) — total ?; splits: test (-1)
• MBPP(+) — total ?; splits: test (-1)
• BigCodeBench — total ?; splits: test (-1)
• LiveCodeBench (V5) — total ?; splits: test (-1)

Metrics

• pass@1 (primary) — range: percent
  • Percentage of problems where the generated code passes all provided unit tests. Evaluated under greedy decoding with a single attempt per problem.

Input / output format

Input: Natural language programming problem description (instruction), sometimes with starter code or function signature.

Output: Python code implementing the solution to the problem.

Scoring recipe

def compute_metric(predictions, gold_tests):
    passed = 0
    for pred, tests in zip(predictions, gold_tests):
        if run_unit_tests(pred, tests):  # returns True if all tests pass
            passed += 1
    return (passed / len(predictions)) * 100

Common pitfalls

• Greedy decoding is used instead of sampling, which may underestimate performance on problems requiring diverse attempts.
• Max generation length is set to 16,384 tokens, which is unusually long and may allow models to output excessive reasoning or boilerplate before code.
• Different evaluation harnesses (EvalPlus vs. Skythought-Evals) are used across benchmarks, meaning test suite coverage and strictness vary between datasets.

Evidence (verbatim from paper)

We evaluate models on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench (V5), each designed to assess different aspects of code generation, including functional correctness, external library usage, and competitive programming challenges. We use EvalPlus for HumanEval(+) and MBPP(+) evaluation, and Skythought-Evals for LiveCodeBench evaluation. We evaluate performance on both Complete and Instruct subsets of BigCodeBench. We follow the official setups in each benchmark and evaluate all models using greedy decoding with a maximum generation length of 16,384 tokens.

Citation

@misc{zhangchen2025kodcode,
  title={KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding},
  author={Zhangchen Xu et al. (2025)},
  year={2025},
  note={arXiv:2503.02951}
}

• arXiv: 2503.02951