Finchart Bench Eval

finchart-bench-eval

FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models — Dong Shu et al. (2025) (arXiv:2507.14823, 2025)

What this evaluates

Evaluates vision-language models' ability to comprehend real-world financial charts. It probes spatial reasoning, instruction following, and factual extraction across True/False, Multiple Choice, and open-ended Question Answering tasks.

Datasets

• FinChart-Bench — total 7016; splits: test (7016)

Metrics

• Exact Match (EM) (primary) — range: percent
  • 1 if the model's extracted answer exactly matches the single-token ground truth, else 0.
• Average (Avg.) score — range: percent
  • Weighted average of task scores: (XScore_TF + YScore_MC + Z*Score_QA) / (X+Y+Z), where X, Y, Z are question counts per task.

Input / output format

Input: Financial chart image paired with a question (True/False, Multiple Choice, or Question Answering) and a formatting instruction.

Output: A single token answer enclosed in the exact string: Result = [[ answer ]]

Scoring recipe

def compute_metrics(predictions, golds, task_types, task_counts):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    task_scores = {}
    for task in ['TF', 'MC', 'QA']:
        task_em = [s for s, t in zip(em_scores, task_types) if t == task]
        task_scores[task] = sum(task_em) / len(task_em) if task_em else 0.0
    avg = (task_counts['TF'] * task_scores['TF'] +
           task_counts['MC'] * task_scores['MC'] +
           task_counts['QA'] * task_scores['QA']) / sum(task_counts.values())
    return avg, task_scores

Common pitfalls

• Ground truth answers are constrained to a single token, which may oversimplify complex financial chart queries.
• Models must strictly adhere to the 'Result = [[ answer ]]' output format; deviations cause automated parsing failures.
• Performance can degrade in newer model versions despite architectural upgrades, indicating instability in chart reasoning.

Evidence (verbatim from paper)

A key motivation behind our benchmark is to eliminate the ambiguity commonly found in existing benchmarks. To this end, we design all ground truth answers to consist of a single token, making Exact Match (EM) an ideal evaluation metric due to its reliability and lack of ambiguity. In addition to EM, we introduce an Average (Avg.) score, which represents the weighted average of the model’s scores across the three tasks, taking into account the number of questions in each.

Citation

@misc{shu2025finchartbench,
  title={FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models},
  author={Dong Shu et al. (2025)},
  year={2025},
  note={arXiv:2507.14823}
}

• arXiv: 2507.14823