Flame Eval

flame-eval

Finance Language Model Evaluation (FLaME) — Matlin et al. (2025) (arXiv:2506.15846, 2025)

What this evaluates

Evaluates foundation and reasoning-reinforced language models across 20 core financial NLP tasks. It probes capabilities in numeric reasoning, entity classification, information retrieval, question answering, and summarization, while also measuring inference efficiency and cost.

Datasets

• FLaME — total ?; splits: test (-1); repo https://github.com/gtfintechlab/FLaME

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall). Used for classification and retrieval tasks.
• BERTScore — range: [0, 1]
  • Similarity metric computed using contextual embeddings from BERT models to compare generated summaries against reference texts.

Input / output format

Input: Task-dependent: text pairs for classification/retrieval, question-answer pairs for QA, and financial documents for summarization.

Output: Task-dependent: discrete class labels, numeric values, ranked lists, or generated text summaries.

Scoring recipe

def compute_f1(preds, golds):
    tp = sum(p == g for p, g in zip(preds, golds))
    fp = sum(p != g and p in golds for p, g in zip(preds, golds))
    fn = sum(g != p and g not in preds for p, g in zip(preds, golds))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• Numeric reasoning tasks suffer sharp performance drops due to requirements for step-by-step deductions and cross-referencing.
• Models may produce off-list label predictions on tasks with large label sets (e.g., Banking77), artificially lowering F1 scores.
• Larger parameter sizes do not strictly guarantee higher performance; cost and throughput must be considered alongside accuracy.

Evidence (verbatim from paper)

Numeric reasoning tasks (like FNXL for numeric labeling or ConvFinQA for multi-step financial statements) remain especially challenging, with F1 scores for FNXL often below 0.06, signaling that even large models struggle to precisely map an extremely large amount of categories to numeric content.

Citation

@misc{matlin2025flame,
  title={Finance Language Model Evaluation (FLaME)},
  author={Matlin et al. (2025)},
  year={2025},
  note={arXiv:2506.15846}
}

• arXiv: 2506.15846