Chart Understanding Eval

chart-understanding-eval

In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding — Fan et al. (2025) (arXiv:2507.14298, 2025)

What this evaluates

Evaluates multimodal language models' ability to comprehend diverse chart types, extract underlying numerical data, and answer questions across varying complexity levels. It distinguishes between OCR-dependent recognition on annotated charts and true data reasoning on unannotated or raw-data-requiring charts.

Datasets

• ChartQA — total ?; splits: human (-1), augmented (-1)
• PlotQA — total ?; splits: test (-1)
• ChartDQA — total ?; splits: test (-1)
• MMC — total ?; splits: VQA (-1)
• ChartX — total ?; splits: VQA (-1)
• Chart-to-Table — total ?; splits: test (-1)
• Chart-to-Text — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match accuracy between the model's generated text answer and the ground truth answer. Computed separately for the human and augmented splits on ChartQA.
• F1 score — range: [0, 1]
  • Token-level F1 score measuring the overlap between predicted and gold table structures/values on Chart-to-Table.
• benchmark-specific metrics — range: varies
  • MMC, ChartX, and ChartDQA follow their respective official evaluation protocols and scoring rules as defined in their original publications.

Input / output format

Input: Chart image paired with a natural language question. For ChartDQA, models may also receive access to raw underlying data or JSON representations.

Output: Natural language text answer or structured JSON, conforming to each benchmark's specific output schema.

Scoring recipe

def evaluate(predictions, gold, benchmark):
    if benchmark == 'ChartQA':
        return exact_match_accuracy(predictions, gold)
    elif benchmark == 'Chart-to-Table':
        return compute_f1_score(predictions, gold)
    else:
        return run_official_evaluator(predictions, gold, benchmark)

Common pitfalls

• High scores on annotated benchmarks (e.g., ChartQA) may primarily reflect OCR capability rather than true chart understanding or data reasoning.
• PlotQA evaluation is strictly zero-shot for most baselines because training them on PlotQA is infeasible, creating an asymmetric comparison.
• Underlying data evaluation (ChartDQA) requires explicit reasoning over raw numerical values, which models often fail at if they rely solely on visual annotations.

Evidence (verbatim from paper)

As shown in Table 2, ChartScope achieves on-par accuracy with the SOTA on ChartQA. Additionally, ChartScope achieves a competitive F1 score on Chart-to-Table... For benchmark details and evaluation metrics, we follow each benchmark’s protocol; please refer to Sec.B for more information.

Citation

@misc{fan2025chartscope,
  title={In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding},
  author={Fan et al. (2025)},
  year={2025},
  note={arXiv:2507.14298}
}

• arXiv: 2507.14298