Finmme Eval

finmme-eval

FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation — Luo et al. (2025) (arXiv:2505.24714, 2025)

What this evaluates

Evaluates financial multi-modal reasoning capabilities of models on chart-based analysis and domain-specific knowledge. It probes perception, analysis, and reasoning across 18 financial domains and 6 asset classes using multiple-choice and computational problems.

Datasets

• FinMME — total 11000; splits: test (-1); repo https://github.com/luo-junyu/FinMME

Metrics

• FinScore (primary) — range: percent
  • FinScore = F * (1 - P_H), where F is the domain-normalized average raw score and P_H is the hallucination penalty rate. Raw score per MCQ is max(0, c/n - i/s), with c=correct selections, n=total options, i=incorrect selections, s=total selections. F averages raw scores within each of the 18 domains, then averages across domains. P_H is the mean ratio of incorrect to total selections across all questions.

Input / output format

Input: Financial charts/images paired with multiple-choice questions (single and multiple answer) and computational problems.

Output: Selected options for MCQs and computed answers for problems.

Scoring recipe

def compute_fin_score(predictions, gold, domains):
    domain_scores = {}
    for k in domains:
        q_scores = []
        for q in domain_questions[k]:
            c = count_correct(predictions[q], gold[q])
            n = total_options(q)
            i = count_incorrect(predictions[q], gold[q])
            s = total_selected(predictions[q])
            raw = max(0, c/n - i/s)
            q_scores.append(raw)
        domain_scores[k] = mean(q_scores)
    F = mean(domain_scores.values())
    P_H = mean([count_incorrect(p, g)/total_selected(p) for p, g in zip(predictions, gold)])
    return F * (1 - P_H)

Common pitfalls

• Multiple-answer questions use a non-standard scoring formula that penalizes over-selection, so standard exact-match or recall metrics will misrepresent performance.
• Domain normalization averages scores within each of the 18 domains first, then averages across domains, preventing domains with more questions from dominating the final score.
• FinScore is multiplicative, meaning a model with high accuracy but high hallucination rate will receive a drastically lower score than one with slightly lower accuracy but higher reliability.

Evidence (verbatim from paper)

We first define the hallucination penalty rate $P_{H}$, which represents the average ratio of incorrect selections across the dataset: $P_{H}=\text{mean}\left(\frac{i}{s}\right)$. The final FinScore combines the domain-normalized score with the hallucination penalty: $\mathcal{F}=F\cdot(1-P_{H})$, where $F$ is the domain-normalized average score across all questions and $P_{H}$ is the hallucination penalty rate.

Citation

@misc{luo2025finmme,
  title={FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2505.24714}
}

• arXiv: 2505.24714