Chartmimic Eval

chartmimic-eval

ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation — Yang et al. (2024) (arXiv:2406.09961, 2024)

What this evaluates

Evaluates large multimodal models' cross-modal reasoning by generating code to reproduce or modify charts based on visual and textual instructions. It tests visual understanding, code generation, and the integration of textual and visual inputs.

Datasets

• ChartMimic — total 4800; splits: test (-1); repo https://github.com/ChartMimic/ChartMimic

Metrics

• Overall (primary) — range: [0, 100]
  • Composite score averaging Low-Level metrics (Text, Layout, Type, Color, Avg) and High-Level metric (GPT-4o evaluation). Scores are on a 0-100 scale.
• Exec. Rate — range: percent
  • Percentage of generated code snippets that execute without errors.

Input / output format

Input: Chart image paired with a textual instruction. For Direct Mimic, the instruction specifies reproducing the chart. For Customized Mimic, it includes the chart plus user-provided customized data.

Output: Generated code to reproduce or modify the chart.

Scoring recipe

def score_chartmimic(predictions, golds):
    exec_success = 0
    for pred in predictions:
        if execute_code(pred):
            exec_success += 1
    exec_rate = (exec_success / len(predictions)) * 100
    
    # Component scores (Text, Layout, Type, Color, GPT-4o) are computed via automated evaluators
    # as detailed in the paper's appendix. Overall is the average of Low-Level and High-Level scores.
    overall = compute_overall_score(predictions, golds)
    return {'Exec. Rate': exec_rate, 'Overall': overall}

Common pitfalls

• Models frequently hallucinate chart elements or misalign visual data with generated code.
• Providing customized data increases the modality processing burden, which can cause performance drops in some models.
• High execution success rate does not guarantee visual fidelity or correct data mapping.

Evidence (verbatim from paper)

We present the main results of 17 LMMs on ChartMimic. Tab. 3 and Tab. 4 show the results on the Direct Mimic and Customized Mimic task, respectively. ... We also include the code execution success rate (Exec. Rate) and model size (Params).

Citation

@misc{yang2024chartmimic,
  title={ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.09961}
}

• arXiv: 2406.09961