Human Eval Cost Eval

human-eval-cost-eval

AI Agents That Matter — Kapoor et al. (2024) (arXiv:2407.01502, 2024)

What this evaluates

This evaluation probes the accuracy-cost tradeoff of AI coding agents by measuring how often generated solutions pass test cases relative to the actual inference cost required. It highlights whether complex agent architectures provide genuine performance gains over simple retry baselines when compute expenses are accounted for.

Datasets

• HumanEval — total 164; splits: test (164)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of problems where at least one generated solution passes all provided test cases.
• cost — range: other
  • Total dollar expenditure for all model API calls and inference steps across all retries and problems.

Input / output format

Input: Problem description and example test cases from the modified HumanEval benchmark.

Output: Generated Python code solution for the problem.

Scoring recipe

total_cost = 0
correct_count = 0
for problem in problems:
    for attempt in range(max_retries):
        code = model.generate(problem)
        total_cost += get_api_cost(code)
        if run_tests(code, problem.test_cases):
            correct_count += 1
            break
accuracy = correct_count / len(problems)

Common pitfalls

• Ignoring inference cost when comparing agent architectures, leading to false claims of SOTA performance.
• Using compute proxies (e.g., token count) instead of actual dollar costs for cost reporting.
• Failing to test against simple retry baselines that can match complex agent accuracy at a fraction of the cost.

Evidence (verbatim from paper)

We run each agent five times and report the mean accuracy and the mean total cost on the 164 HumanEval problems.

Citation

@misc{kapoor2024aiagents,
  title={AI Agents That Matter},
  author={Kapoor et al. (2024)},
  year={2024},
  note={arXiv:2407.01502}
}

• arXiv: 2407.01502