Chartnet Eval

chartnet-eval

ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding — Kondic et al. (2026) (arXiv:2603.27064, 2026)

What this evaluates

Evaluates multimodal vision-language models on chart understanding tasks, including reconstructing plotting code from charts, extracting tabular data, summarizing chart content, and answering complex reasoning questions.

Datasets

• ChartNet Evaluation Set — total 2000; splits: test (2000)

Metrics

• ChartNet Evaluation Metrics (primary) — range: percent
  • Exec.: fraction of generated scripts executing without error. Code-D: numeric correspondence between plotted values and ground truth. Code-S: structural/syntactic overlap. Img.: visual alignment. Data Extraction: CSV similarity score. Summarization: holistic GPT-4o judge score. QA: RapidFuzz fuzzy accuracy on extracted .

Input / output format

Input: Chart image I; for reconstruction tasks, ground-truth plotting code/data table/summary may be provided as context or reference; for QA, a complex reasoning question paired with I.

Output: For reconstruction: executable plotting script C′. For extraction: CSV table. For summarization: textual summary. For QA: and sections.

Scoring recipe

# 1. Chart Reconstruction
exec_rate = sum(1 for c in preds if run_script(c) == 0) / len(preds)
code_d, code_s, img_sim = gpt4o_judge(preds, gold_code, gold_img)
# 2. Data Extraction
data_sim = gpt4o_judge_csv(preds, gold_csv)
# 3. Summarization
summary_score = gpt4o_judge_text(preds, gold_summary)
# 4. QA
answer = extract_tag(preds, "<answer>")
qa_acc = rapidfuzz.fuzz.ratio(answer, gold_qa) / 100.0

Common pitfalls

• Most metrics (except QA) rely on GPT-4o as an automated judge, which may introduce bias or inconsistency compared to exact programmatic metrics.
• Custom metrics like Code-D, Code-S, and Img. similarity lack standardized definitions and depend heavily on the specific GPT-4o prompt templates in Appendix B.4.
• The 'holistic score' for summarization combines multiple dimensions (coverage, faithfulness, correctness, clarity) into a single number without transparent weighting.

Evidence (verbatim from paper)

To rigorously evaluate the tasks in the core ChartNet dataset, we curate a held-out evaluation suite randomly drawn from ChartNet’s synthetic corpus. The set comprises 2000 chart tuples... We evaluate (a) execution rate (Exec.) — the fraction of generated scripts C′ that execute without error, (b) data fidelity (Code-D) — the correspondence between plotted numeric values and the data defined in ground-truth code, (c) code similarity (Code-S) — the structural and syntactic overlap between generated, C′, and source code, C, and (d) rendered image similarity (Img.) — the visual alignment between the rendered prediction and the input chart I... We report average fuzzy accuracy. All metrics are automatically computed using GPT-4o as a judge, except for the Chart QA with CoT Reasoning task.

Citation

@misc{kondic2026chartnet,
  title={ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding},
  author={Kondic et al. (2026)},
  year={2026},
  note={arXiv:2603.27064}
}

• arXiv: 2603.27064