Livecodebench Eval

livecodebench-eval

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code — Jain et al. (2024) (arXiv:2403.07974, 2024)

What this evaluates

Evaluates large language models' ability to generate, repair, execute, and predict outputs for code across multiple algorithmic and competitive programming problem types. It provides a holistic, contamination-free assessment by continuously updating with new problems and testing models across four distinct coding scenarios.

Datasets

• LiveCodeBench — total 511; splits: test (511)

Metrics

• PASS@1 (primary) — range: [0, 1]
  • Calculates the fraction of correct programs or answers out of 10 generated candidates per problem. Correctness is verified by passing all provided tests for generation/repair scenarios, or via execution-based equivalence checks for execution and test output prediction scenarios.

Input / output format

Input: Problem statement, function signature, and test inputs; for self-repair, includes previously generated code and specific error feedback (syntax, runtime, wrong answer, time-limit); prompts vary by scenario (zero-shot, few-shot, or chain-of-thought).

Output: Generated or repaired source code, predicted execution output, or completed assertion statements, depending on the scenario.

Scoring recipe

def score_pass_at_1(predictions, gold_tests, gold_output, gold_answer, scenario):
    correct = 0
    for pred in predictions:  # 10 samples
        if scenario in ['code_generation', 'self_repair']:
            if all(run_test(pred, t) for t in gold_tests):
                correct += 1
        elif scenario == 'code_execution':
            if execute(pred) == gold_output:
                correct += 1
        elif scenario == 'test_output_prediction':
            if parse_answer(pred) == gold_answer:
                correct += 1
    return correct / len(predictions)

Common pitfalls

• Models trained before a problem's release date may show artificially inflated scores if the problem leaked into their training data (contamination).
• Base models require a one-shot prompt with a fixed example, while instruction-tuned models use zero-shot prompts; applying the wrong prompt format breaks evaluation.
• Self-repair scenarios require injecting the exact error type (syntax, runtime, wrong answer, time-limit) into the prompt; mismatched feedback causes unfair failures.

Evidence (verbatim from paper)

We use the PASS@1 (Kulal et al., 2019; Chen et al., 2021) metric for our evaluations. Specifically, we generate 10 candidate answers for each problem either using API or using vLLM (Kwon et al., 2023). We use nucleus sampling with temperature 0.2 and top-p 0.95 and calculate the fraction of programs or answers that are correct. For the code generation and self-repair scenarios, we use tests to verify the correctness of the programs. For these scenarios, programs must pass all tests to be considered correct. For the code execution scenario, we use an execution-based correctness metric between the generated output and the ground truth output. For the test output prediction scenario, we parse the generated response to extract the answer and use equivalence checks for grading as specified in Section 2.

Citation

@misc{jain2024livecodebench,
  title={LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code},
  author={Jain et al. (2024)},
  year={2024},
  note={arXiv:2403.07974}
}

• arXiv: 2403.07974