Humaneval V Eval

humaneval-v-eval

HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks — Zhang et al. (2024) (arXiv:2410.12381, 2024)

What this evaluates

This benchmark evaluates large multimodal models' ability to perform high-level visual reasoning over complex diagrams in coding contexts. It specifically probes spatial transformations, topological relationships, and dynamic pattern understanding by requiring models to translate visual information into executable code.

Datasets

• HumanEval-V — total 253; splits: test (253); repo https://github.com/HumanEval-V/HumanEval-V-Benchmark

Metrics

• pass@1 (primary) — range: percent
  • The fraction of tasks for which the single generated code solution passes all provided test cases. Calculated as (number of passed tasks) / (total tasks).

Input / output format

Input: A complex diagram image and a task prompt describing the coding objective. In iterative evaluation pipelines, the input also includes execution feedback such as syntax error messages or failed test case inputs and expected outputs.

Output: Executable code (typically Python) that solves the diagram-based task. In the V2T2C pipeline, an intermediate textual description of the diagram is generated first before code implementation.

Scoring recipe

def compute_pass_at_1(predictions, test_cases):
    passed = 0
    for code in predictions:
        try:
            if run_code_and_check_tests(code, test_cases):
                passed += 1
                break
        except Exception:
            pass
    return passed / len(predictions)

Common pitfalls

• Evaluating direct V2C (Visual-to-Code) pipelines may conflate visual comprehension with coding ability; the paper recommends decoupling via V2T2C to isolate visual reasoning.
• Models may appear to improve via iterative refinement, but evaluators must verify that corrections stem from genuine diagram understanding rather than hardcoding exposed test cases.
• Long chain-of-thought reasoning can hit token limits (e.g., 20k tokens), causing valid reasoning to fail simply due to truncation.

Evidence (verbatim from paper)

For the iterative evaluation, we select the most capable LMMs across different parameter scales, using greedy decoding and the pass@1 metric. Figure 9 presents the results, where iter 0 represents the first round of generation without feedback. We observe that LMMs generally improve across iterations, with more capable models achieving larger performance gains.

Citation

@misc{zhang2024humanevalv,
  title={HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.12381}
}

• arXiv: 2410.12381