Mme Finance Eval

mme-finance-eval

MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning — Gan et al. (2024) (arXiv:2411.03314, 2024)

What this evaluates

Evaluates multimodal large language models' ability to understand and reason over financial charts, tables, and documents. It probes fine-grained visual perception, spatial reasoning, numerical calculation, and complex financial decision-making in a domain-specific context.

Datasets

• MME-Finance — total 2274; splits: test (-1)

Metrics

• LLM-based score (0-5) (primary) — range: percent
  • An LLM evaluator assigns a score from 0 to 5 per sample. This score is multiplied by 20% to be normalized to a 0-100 scale.

Input / output format

Input: Image, question, ground truth answer, and the model's generated response.

Output: A normalized score from 0 to 100 (derived from an LLM judge's 0-5 rating).

Scoring recipe

def compute_metric(predictions, gold):
    scores = []
    for img, q, gt, pred in zip(predictions['images'], predictions['questions'], gold['ground_truth'], predictions['responses']):
        prompt = f'Image: {img}, Question: {q}, Ground Truth: {gt}, Model Response: {pred}'
        raw_score = llm_evaluator.generate(prompt) # returns 0-5
        scores.append(raw_score * 20)
    return sum(scores) / len(scores)

Common pitfalls

• Relies on an LLM-as-a-judge, which may introduce bias or inconsistency compared to human experts.
• The 'Not Applicable' (NA) task explicitly prompts models to opt-out, potentially lowering the measured hallucination rate compared to standard open-ended evaluation.
• Performance on mobile photographs is heavily influenced by image resolution and capture angle, conflating visual quality with reasoning capability.

Evidence (verbatim from paper)

We fill the prompt template with image, question, ground truth, and response from an MLLM, and take the filled prompt into an LLM-based evaluator for generating a score range from 0 to 5 for one sample. The scores are multiplied by 20% to be normalized.

Citation

@misc{gan2024mmefinance,
  title={MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning},
  author={Gan et al. (2024)},
  year={2024},
  note={arXiv:2411.03314}
}

• arXiv: 2411.03314