Codearena Eval

codearena-eval

Evaluating and Aligning CodeLLMs on Human Preference — Jian Yang et al. (2024) (arXiv:2412.05210, 2024)

What this evaluates

Evaluates code LLMs' alignment with human preferences on real-world, non-algorithmic coding tasks. It measures how well model-generated code matches human-like quality and preference compared to a baseline, rather than just syntactic or execution correctness.

Datasets

• CodeArena — total ?; splits: test (-1)
• EvalPlus — total ?; splits: test (-1)
• MultiPL-E — total ?; splits: test (-1)

Metrics

• Pass@1 (primary) — range: percent
  • Greedy Pass@1 measures the fraction of generated solutions that pass all provided test cases. The expected function is extracted from the model output and executed against the test suite to verify correctness.
• Win rate (primary) — range: percent
  • Pairwise preference score calculated by an LLM judge (GPT-4o) comparing a model response against a baseline. Each question is scored twice with swapped input orderings (A,B and B,A) to mitigate position bias, and the win rate is averaged across these runs.

Input / output format

Input: For code execution benchmarks (EvalPlus, MultiPL-E): a coding problem description and associated test cases. For preference benchmark (CodeArena): a coding question and two model-generated code responses (baseline and target model).

Output: For code execution benchmarks: a generated code snippet containing the expected function. For preference benchmark: a judgment indicating which response is better or a tie.

Scoring recipe

def score_pass_at_1(predictions, test_cases):
    correct = 0
    for pred in predictions:
        func = extract_function(pred)
        if run_tests(func, test_cases):
            correct += 1
    return correct / len(predictions)

def score_win_rate(predictions, baseline, question):
    j1 = llm_judge(question, baseline, predictions)
    j2 = llm_judge(question, predictions, baseline)
    wins = sum(1 for j in [j1, j2] if j == 'model_wins')
    return wins / 2.0

Common pitfalls

• CodeArena tasks are non-algorithmic and cannot be evaluated via automated code execution; they strictly require pairwise LLM judging.
• The LLM judge's win rate must be averaged over two different input orderings (A,B and B,A) to account for position bias, not just a single comparison.
• Pass@1 is computed using greedy decoding, not sampling, which may underestimate performance if diverse solutions are needed.

Evidence (verbatim from paper)

Different from the EvalPlus and MultiPL-E, CodeArena consists of many non-algorihtmic, which is not suitable for code-execution-based evaluation. Each question is scored twice to calculate the win rate and tie rate by GPT-4o using a different input order “A, B” and “B, A”, where “A” is the baseline from gpt-4-turbo-2024-04-09 and “B” is the model-generated response. Given the model-generated response, we extract the expected function and feed the test cases into the extracted function to verify the correctness of the generation. We adopt greedy Pass@1 to report the results on EvalPlus and MultiPL-E.

Citation

@misc{yang2024codearena,
  title={Evaluating and Aligning CodeLLMs on Human Preference},
  author={Jian Yang et al. (2024)},
  year={2024},
  note={arXiv:2412.05210}
}

• arXiv: 2412.05210