Cfbenchmark Mm Eval

cfbenchmark-mm-eval

CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model — Jiangtong Li et al. (2025) (arXiv:2506.13055, 2025)

What this evaluates

Evaluates multimodal large language models' ability to interpret financial charts, tables, and diagrams in Chinese, and answer domain-specific questions. It probes visual reasoning, statistical and structural analysis, and financial concept comprehension under zero-shot conditions.

Datasets

• CFBenchmark-MM — total 9356; splits: test (9356)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered objective (multiple-choice) questions. Calculated as the number of exact-match correct predictions divided by the total number of objective questions.
• subjective_score — range: percent
  • Percentage score awarded to short-answer responses based on point-by-point correctness against reference answers. Evaluated via human or LLM judgment on a 0-100 scale.

Input / output format

Input: A Chinese question {question}, optionally accompanied by a financial chart/image and a textual caption {caption}. The prompt instructs the model to examine chart components (axes, legends, data points) and answer either as a multiple-choice selection or a point-by-point short answer.

Output: For multiple-choice questions: a list of all correct option letters. For short-answer questions: a structured, point-by-point textual response addressing each part of the query.

Scoring recipe

def calc_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if set(p) == set(g))
    return correct / len(golds) * 100

def calc_subjective_score(predictions, golds):
    # Score based on point-by-point alignment with reference answers
    # Typically 0-100 scale as reported in paper
    scores = [score_response(p, g) for p, g in zip(predictions, golds)]
    return sum(scores) / len(scores)

Common pitfalls

• Models frequently misinterpret visual elements like axes, legends, or data points, leading to incorrect numerical extraction.
• Adding generated captions to the input can degrade performance for some models (e.g., GPT-4V) due to redundant or conflicting information.
• Financial concept misunderstandings (e.g., confusing compound vs. average growth rates) persist even when models correctly read the chart.
• Evaluation excludes models lacking Chinese language support, which may bias open-source model comparisons.

Evidence (verbatim from paper)

However, even advanced MLLMs like GPT-4V achieve only 52% accuracy and 38% score in objective and subjective questions, falling short of real-world application requirements.

Citation

@misc{li2025cfbenchmarkmm,
  title={CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model},
  author={Jiangtong Li et al. (2025)},
  year={2025},
  note={arXiv:2506.13055}
}

• arXiv: 2506.13055