Chartom Eval

chartom-eval

CHARTOM: A Visual Theory-of-Mind Benchmark for LLMs on Misleading Charts — Bharti et al. (2024) (arXiv:2408.14419, 2024)

What this evaluates

This benchmark evaluates large language models' ability to comprehend factual data in charts (FACT task) and their capacity to predict how visual manipulations mislead human readers (MIND task). It probes visual theory-of-mind by measuring whether models can distinguish between objective chart data and subjective human perceptual biases introduced by deceptive visualization techniques.

Datasets

• CHARTOM — total 112; splits: test (112)

Metrics

• FACT_accuracy (primary) — range: [0, 1]
  • Accuracy of the model's answer compared to the ground-truth answer for the chart comprehension question. Supports multiple-choice, free-text, and ranking formats.
• HMI — range: [0, 1]
  • Mean Absolute Error between the model's predicted decimal fraction (0–1) and the ground-truth Human Misleadingness Index (HMI) derived from human experiments.

Input / output format

Input: An image of a chart (line, bar, pie, scatter, or map) accompanied by a text prompt containing either a FACT question (chart comprehension) or a MIND question (predicting human misleadingness).

Output: For FACT: the selected answer (choice, text string, or ranked list). For MIND: a decimal number between 0 and 1 representing the predicted fraction of misled humans, followed by a textual justification.

Scoring recipe

def score_chartom(predictions, golds):
    fact_accs = []
    hmi_errors = []
    for pred, gold in zip(predictions, golds):
        if gold['type'] == 'FACT':
            fact_accs.append(1.0 if pred['answer'] == gold['answer'] else 0.0)
        elif gold['type'] == 'MIND':
            pred_val = float(pred['prediction_decimal'])
            hmi_val = float(gold['hmi'])
            hmi_errors.append(abs(pred_val - hmi_val))
    return {
        'FACT_accuracy': sum(fact_accs) / len(fact_accs),
        'HMI': sum(hmi_errors) / len(hmi_errors)
    }

Common pitfalls

• Models often conflate the FACT task (extracting true data values) with the MIND task (predicting human perceptual bias), leading to incorrect predictions on manipulated charts.
• The MIND output requires a specific format: a decimal prediction first, then a justification. Models that output only text or reverse the order will fail strict parsing.
• HMI ground truth is derived from human experiments, not from the chart's visual properties alone; evaluators must use the provided HMI values rather than estimating misleadingness themselves.

Evidence (verbatim from paper)

Each chart comes with a MIND question, and the answer key is the Human Misleadingness Index (HMI, see Section[4.2]) obtained from human experiments. What fraction of typical university students do you predict will be misled by the chart when answering the question? First, give your prediction as a decimal number between 0 and 1, then justify your prediction in words.

Citation

@misc{bharti2024chartom,
  title={CHARTOM: A Visual Theory-of-Mind Benchmark for LLMs on Misleading Charts},
  author={Bharti et al. (2024)},
  year={2024},
  note={arXiv:2408.14419}
}

• arXiv: 2408.14419