Plot2code Eval

plot2code-eval

Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots — Chengyue Wu et al. (2024) (arXiv:2405.07990, 2024)

What this evaluates

Evaluates multi-modal large language models' ability to visually interpret scientific plots and generate corresponding executable Python (matplotlib) code. It probes fine-grained visual reasoning, text extraction from dense plots, and precise code generation for data visualization.

Datasets

• Plot2Code — total 132; splits: test (-1)

Metrics

• Pass Rate (primary) — range: percent
  • Percentage of generated code snippets that execute without errors and produce a valid plot.
• Text-Match — range: percent
  • Ratio measuring the textual similarity between the generated plot and the reference plot, typically computed via string or embedding overlap.
• Rating — range: [0, 10]
  • GPT-4V overall fidelity rating on a scale of 0 to 10, assessing how closely the generated image matches the reference plot visually.

Input / output format

Input: A scientific plot image (matplotlib figure). In the 'Conditional Asking' setting, an additional text instruction describing the plot is appended.

Output: Executable Python code using matplotlib to reproduce the plot.

Scoring recipe

def score(predictions, golds):
    pass_count = 0
    text_matches = []
    ratings = []
    for pred, gold in zip(predictions, golds):
        if execute_code(pred):
            pass_count += 1
        text_matches.append(calculate_text_similarity(pred, gold))
        ratings.append(gpt4v_judge(pred, gold))
    return {
        'Pass Rate': pass_count / len(predictions) * 100,
        'Text-Match': sum(text_matches) / len(text_matches),
        'Rating': sum(ratings) / len(ratings)
    }

Common pitfalls

• Models heavily rely on textual instructions rather than visual reasoning, causing pass rates to drop when instructions are added (Conditional Asking).
• Chain-of-Thought and Plan-and-Solve prompting strategies do not consistently improve performance over default prompts on this benchmark.
• High-resolution image inputs significantly boost performance, indicating standard low-resolution encoders miss fine plot details.

Evidence (verbatim from paper)

We showcase the quantitative results of (M)LLMs on our Plot2Code benchmark here. The code pass rate, text-match ratio, and GPT-4V overall rating for both direct asking and conditional asking scenarios are reported in Table 3.

Citation

@misc{wu2024plot2code,
  title={Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots},
  author={Chengyue Wu et al. (2024)},
  year={2024},
  note={arXiv:2405.07990}
}

• arXiv: 2405.07990