Famma Eval

famma-eval

FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering — Xue et al. (2024) (arXiv:2410.04526, 2024)

What this evaluates

Evaluates multimodal large language models on financial domain reasoning, including calculation-heavy arithmetic problems and knowledge-intensive non-arithmetic questions. It probes cross-lingual capabilities and robustness to data contamination by testing on both textbook-derived and expert-crafted live questions.

Datasets

• FAMMA-Basic — total 1945; splits: test (1945); repo https://github.com/famma-bench/bench-script
• FAMMA-LivePro — total 103; splits: test (103); repo https://github.com/famma-bench/bench-script

Metrics

• accuracy (primary) — range: percent
  • Average percentage of questions where the model's first generated response is judged correct by an LM evaluator (GPT-4o at temperature 0). Responses are categorized as correct or incorrect, and the score is computed over the entire set.

Input / output format

Input: Multimodal prompts containing financial questions, text, and images/tables/charts. For text-only models, images and tables are pre-processed into plain text using PaddleOCR. Evaluated in a zero-shot setting with consistent instruction prompts.

Output: Free-form text response. For multiple-choice questions, the model outputs a lettered option, which is extracted via regex or GPT-4o. For open-ended questions, the model outputs a direct answer or reasoning trace.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        is_correct = lm_evaluator_gpt4o(pred, gold) == "correct"
        if is_correct:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• OCR preprocessing for text-only models can introduce transcription errors, artificially lowering multimodal performance.
• Multiple-choice answer extraction relies on regex/GPT-4o, which may fail if models deviate from expected formatting.
• LM-based evaluation (GPT-4o) replaces exact match, potentially introducing evaluator bias or inconsistency despite temperature=0.

Evidence (verbatim from paper)

During the evaluation process, we use GPT-4o as an LM evaluator to assess the accuracy of responses generated by LLMs for each question. The reported score represents the accuracy of these responses. Each response is categorized as either correct or incorrect, and the reported score reflects the average accuracy across the entire set of questions.

Citation

@misc{xue2024famma,
  title={FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering},
  author={Xue et al. (2024)},
  year={2024},
  note={arXiv:2410.04526}
}

• arXiv: 2410.04526