qhjqhj00 finmr-eval
Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach — Deng et al. (2025) (arXiv:2506.06282, 2025)
What this evaluates
Evaluates the multimodal financial reasoning capabilities of LLMs and MLLMs on expert-level question-answer pairs spanning 15 financial domains. It probes the models' ability to interpret complex visual data (charts, tables), apply domain-specific formulas, and perform multi-step logical calculations.
Datasets
- FinMR — total 3200; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) * 100. Reported at the overall level, for sub-categories (Math, Expertise), and across 15 specific financial domains.
Input / output format
Input: A question paired with an image (e.g., financial chart or table) or an image caption. For text-only LLMs, images are converted to captions using GPT-4o.
Output: A final answer, typically generated via Chain-of-Thought (CoT) or Error Feedback Learning (EFL) prompting. The model outputs reasoning steps followed by a definitive answer.
Scoring recipe
def calculate_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
if pred.strip() == gold.strip():
correct += 1
return (correct / len(gold_answers)) * 100
Common pitfalls
- Confusing performance gains from GPT-4o-generated image captions with true multimodal reasoning, as captions artificially boost LLM scores without actual visual processing.
- Overlooking the 'answer not found' error type, where models fail to produce a final answer due to token repetition, which should be counted as incorrect.
- Failing to distinguish between mathematical reasoning and expertise reasoning tasks, as they require fundamentally different capabilities (logical rigor vs. contextual understanding).
Evidence (verbatim from paper)
In particular, the textual LLM DeepSeek-R1 and multimodal Gemini-1.5-Pro gained 71.88% and 82.06% overall accuracy, respectively. In contrast, open-source models such as LLaMa 3.2 and LLaVa-NEXT demonstrate significantly lower overall performances, with accuracies falling below 30%.
Citation
@misc{deng2025finmr,
title={Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach},
author={Deng et al. (2025)},
year={2025},
note={arXiv:2506.06282}
}
- arXiv: 2506.06282