Finset Eval

finset-eval

FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models — Bhatia et al. (2024) (arXiv:2402.10986, 2024)

What this evaluates

Evaluates financial LLMs across seven text-based tasks (sentiment analysis, NER, number understanding, summarization, stock movement prediction, credit scoring, firm disclosure) and three multimodal/hallucination tasks (ChartQA, FinVQA, FinTerms). It measures domain-specific reasoning, instruction following, and hallucination mitigation in financial contexts.

Datasets

• FinSet — total ?; splits: test (-1); repo https://github.com/UBC-NLP/fintral
• ChartQA — total ?; splits: test (-1)
• FinVQA — total ?; splits: test (-1)
• FinTerms-MCQ — total ?; splits: test (-1)
• FinTerms-Gen — total 150; splits: test (150)
• Finance Bench — total 150; splits: test (150)

Metrics

• Task Accuracy/F1 (primary) — range: [0, 1]
  • Reported as decimal scores (0.00–1.00) for SA, NER, NU, TS, SMP, CS, and FD. The exact metric (accuracy vs. F1) is not specified in the text.
• Hallucination Index (HI) — range: [0, 1]
  • Proportion of correctly generated definitions in the FinTerms-MCQ multiple-choice task. Calculated as correct predictions divided by total questions.
• Human Evaluation — range: other
  • Responses categorized into four quality tags (A: correct/satisfying, B: acceptable, C: significant errors, D: irrelevant) by two finance experts.
• Finance Bench Accuracy — range: percent
  • Percentage of correct, incorrect, and failed responses on open-book financial QA.

Input / output format

Input: Text prompts for seven financial tasks (SA, NER, NU, TS, SMP, CS, FD); image+text prompts for ChartQA and FinVQA; multiple-choice questions for FinTerms-MCQ; open-book financial QA questions for Finance Bench.

Output: Task-specific outputs: labels/entities/numbers/summaries/movement directions/scores/disclosures; chart descriptions/answers; selected MCQ option; free-form QA answers.

Scoring recipe

# Text tasks (SA, NER, NU, TS, SMP, CS, FD)
pred = model.generate(prompt)
score = compute_task_metric(pred, gold) # metric unspecified (acc/f1)

# Hallucination Index (FinTerms-MCQ)
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
HI = correct / len(gold)

# Human Evaluation (FinTerms-Gen)
labels = human_annotate(predictions) # A, B, C, D
# Reported as distribution across categories

# Finance Bench
correct = sum(1 for p, g in zip(predictions, gold) if is_correct(p, g))
accuracy = correct / len(gold)

Common pitfalls

• The exact evaluation metric (accuracy vs. F1) for the seven text tasks is not explicitly stated in the text, only decimal scores are reported.
• FinTerms-Gen human evaluation only uses 128 out of 150 Q&A pairs where both annotators agreed, which may introduce selection bias.
• Finance Bench evaluation uses only the 150-question open-source sample, not the full 10,231-question proprietary dataset.

Evidence (verbatim from paper)

We derive a hallucinations index (HI), defined as the proportion of correctly generated definitions by each model (higher is better), based on the models’ performance in this MCQ task. As seen in Table [8], the models’ performances on the HI vary significantly.

Citation

@misc{bhatia2024fintral,
  title={FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models},
  author={Bhatia et al. (2024)},
  year={2024},
  note={arXiv:2402.10986}
}

• arXiv: 2402.10986