Human Eval Functional Accuracy

human-eval-functional-accuracy

Capturing Failures of Large Language Models via Human Cognitive Biases — Jones et al. (2022) (arXiv:2202.12299, 2022)

What this evaluates

Evaluates a code generation model's ability to produce correct, executable Python functions from docstrings and function signatures. It measures whether the generated code passes all provided unit tests for each programming problem.

Datasets

• HumanEval — total 164; splits: test (164); repo https://github.com/openai/human-eval

Metrics

• functional accuracy (primary) — range: [0, 1]
  • The fraction of programs that pass all of the provided test cases. Calculated as (number of problems where generated code passes all tests) / (total number of problems).

Input / output format

Input: Function signature and docstring describing the desired functionality, optionally with prepended irrelevant preceding functions or anchor functions containing similar but incorrect code.

Output: Python code completion (function body) generated via greedy decoding.

Scoring recipe

def compute_functional_accuracy(generated_code, test_cases):
    passed = 0
    for code, tests in zip(generated_code, test_cases):
        try:
            exec(code)
            if all(test() for test in tests):
                passed += 1
        except Exception:
            pass
    return passed / len(generated_code)

Common pitfalls

• Models may output the prepended framing or anchor line verbatim instead of solving the problem, artificially lowering accuracy without reflecting true reasoning failure.
• Prepending anchor functions can sometimes lead to correct solutions that incorporate the anchor's logic, meaning accuracy drops do not always indicate a complete failure to solve the task.
• Greedy decoding is used, which may not reflect the model's peak capability compared to sampling-based decoding.

Evidence (verbatim from paper)

Following Chen et al. (2021), we measure performance on HumanEval with functional accuracy: the fraction of programs that pass all of the test cases provided at the url: https://github.com/openai/human-eval.

Citation

@misc{jones2022capturing,
  title={Capturing Failures of Large Language Models via Human Cognitive Biases},
  author={Jones et al. (2022)},
  year={2022},
  note={arXiv:2202.12299}
}

• arXiv: 2202.12299