Chartassistant Eval

chartassistant-eval

ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning — Fanqing Meng et al. (2024) (arXiv:2401.02384, 2024)

What this evaluates

Evaluates a multimodal language model's ability to comprehend, summarize, and answer questions about various chart types (base and specialized). It probes chart-to-text generation, open-ended and numerical question answering, referring question answering, and chart-to-table translation.

Datasets

• ChartQA — total ?; splits: aug (-1), human (-1)
• Chart-to-Text — total ?; splits: Pew (-1), Statista (-1)
• OpenCQA — total ?; splits: test (-1)
• MathQA — total ?; splits: test (-1)
• ReferQA — total ?; splits: test (-1)
• RealQA — total ?; splits: Math (-1), Extract (-1)

Metrics

• relaxed_correctness (primary) — range: percent
  • Exact match with a 5% numerical tolerance. A prediction is correct if it matches the gold answer exactly, or if the absolute relative error between the predicted and gold numerical values is ≤ 0.05.
• BLEU — range: percent
  • Standard n-gram based BLEU score used for chart summarization and open-ended QA generation tasks.
• RMS_F1 — range: percent
  • Root Mean Square F1 score used to evaluate chart-to-table translation performance, following the DePlot protocol.

Input / output format

Input: Chart image paired with a natural language question or instruction (e.g., summarization prompt, numerical QA, referring QA, or table extraction request).

Output: Text response containing the answer, summary, or structured table data.

Scoring recipe

def compute_relaxed_correctness(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            p, g = float(pred), float(gold)
            if abs(p - g) / max(abs(g), 1e-6) <= 0.05:
                correct += 1
        except ValueError:
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
    return correct / len(predictions)

Common pitfalls

• Relaxed correctness allows a 5% numerical tolerance, so strict exact-match evaluation will incorrectly penalize valid answers.
• BLEU evaluation on Chart-to-Text and OpenCQA is highly sensitive to reference wording due to limited ground-truth references, making scores volatile and heavily dependent on reference alignment.
• Baseline models are fine-tuned on the training split of each test dataset, whereas ChartAssistant is evaluated after a single unified training phase, creating an unfair comparison if not explicitly accounted for.

Evidence (verbatim from paper)

Metrics. For evaluating ChartQA, MathQA, and ReferQA, we adopt the approach used in previous studies [25, 31], which considers relaxed correctness (allowing for an exact match with tolerance for a 5% numerical error). As for Chart-to-Text and OpenCQA, we employ BLEU as the evaluation metric following previous works [25, 31]. For chart-to-table translation, we use RMS_F1 from DePlot [24].

Citation

@misc{meng2024chartassistant,
  title={ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning},
  author={Fanqing Meng et al. (2024)},
  year={2024},
  note={arXiv:2401.02384}
}

• arXiv: 2401.02384