Open Finllm Leaderboard Eval

open-finllm-leaderboard-eval

Evaluation and Benchmarking Suite for Financial Large Language Models and Agents — Lin et al. (2026) (arXiv:2602.19073, 2026)

What this evaluates

Evaluates the capability of financial LLMs and agents across seven core financial task categories, including information extraction, sentiment analysis, question answering, text generation, risk management, forecasting, and decision-making. The benchmark aggregates 42 existing financial datasets to provide a standardized comparison of model performance and compliance readiness.

Datasets

• Open FinLLM Leaderboard (42 financial datasets) — total ?; splits: test (-1)

Metrics

• average score across all tasks (primary) — range: other
  • Not explicitly defined in the provided text; described as an aggregate of per-task scores computed across the 42 datasets organized into seven categories.
• Model Openness Framework (MOF) class — range: other
  • A compliance rating (Class I, II, or III) that evaluates model licensing and training data transparency to mitigate open-washing risks.

Input / output format

Input: Multimodal financial data and prompts corresponding to the 42 datasets across seven task categories (IE, TA, QA, TG, RM, FO, DM).

Output: Task-specific predictions or generated text, plus an MOF openness class rating (Class I, II, or III).

Scoring recipe

scores = []
for dataset in datasets_42:
    pred = model.predict(dataset)
    scores.append(compute_task_metric(pred, dataset.gold))
overall_score = mean(scores)
openness_class = evaluate_openness(model_license, training_data)
return overall_score, openness_class

Common pitfalls

• The benchmark aggregates 42 pre-existing datasets rather than defining a single unified test set, so results are highly task-dependent and not directly comparable across categories.
• The 'openness class' metric evaluates licensing and data transparency compliance, not model capability, and is scored separately from task accuracy.
• Exact evaluation formulas, dataset splits, and versioning are not provided in this section; they are hosted externally on the Hugging Face space and documentation site.

Evidence (verbatim from paper)

The current $42$ financial datasets are organized into seven categories, as given in Table [1]. The leaderboard on Huggingface is designed to show transparent and comprehensive evaluation results for financial LLMs. Leaderboard Table. The central part is a table containing: rank, model name, average score across all tasks, openness class (Class I, II, III under MOF in Section [5.3])

Citation

@misc{lin2026finllmleaderboard,
  title={Evaluation and Benchmarking Suite for Financial Large Language Models and Agents},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2602.19073}
}

• arXiv: 2602.19073