Climateviz Eval

climateviz-eval

ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts — Su et al. (2025) (arXiv:2506.08700, 2025)

What this evaluates

This benchmark evaluates multimodal models' ability to perform statistical reasoning and fact verification on scientific charts. It tests whether models can correctly classify claims as supporting, refuting, or not enough information (NEI) based on visual data, and assesses the quality of their generated structured explanatory triplets.

Datasets

• ClimateViz — total 49862; splits: train (-1), dev (-1), test (-1); repo https://github.com/Albasu120491/ClimateViz

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted fact-checking labels (support, refute, NEI) out of the total number of instances.
• macro F1 — range: [0, 1]
  • Unweighted mean of recall or precision for each class (support, refute, NEI), treating all classes equally regardless of frequency.
• BLEU — range: [0, 1]
  • N-gram overlap between generated explanatory triplets and gold triplets, penalized for brevity.
• METEOR — range: [0, 1]
  • Harmonic mean of unigram precision and recall, weighted by alignment and stemming.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated and gold explanatory triplets.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of generated and gold triplets using BERT.

Input / output format

Input: Chart image, chart caption, and claim text. In the CTT setting, a structured table extracted from the chart is also provided.

Output: Either a single fact-checking label (support, refute, or NEI), or a pair consisting of structured explanatory triplets and the final label.

Scoring recipe

def compute_metrics(predictions, golds):
    # predictions, golds: lists of labels ('support', 'refute', 'NEI')
    acc = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    f1 = macro_f1_score(golds, predictions)
    return {'accuracy': acc, 'macro_f1': f1}

Common pitfalls

• Confusing the CT (Chart+Text) and CTT (Chart+Table+Text) input configurations, which produce divergent performance trends across models.
• Assuming few-shot prompting universally improves results; the paper shows it can significantly degrade accuracy for models like Gemini 2.5 and GPT-4o.
• Evaluating explanation-augmented outputs solely on label accuracy, while ignoring the required BLEU, METEOR, ROUGE-L, and BERTScore metrics for triplet quality.

Evidence (verbatim from paper)

We evaluate label classification using accuracy and macro F1. We further evaluate generated triplets using BLEU (Papineni et al., [2002]), METEOR (Banerjee and Lavie, [2005]), ROUGE-L (Lin, [2004]) and BERTScore (Zhang et al., [2020]).

Citation

@misc{su2025climateviz,
  title={ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2506.08700}
}

• arXiv: 2506.08700