Financebench Eval

financebench-eval

FinanceBench: A New Benchmark for Financial Question Answering — Islam et al. (2023) (arXiv:2311.11944, 2023)

What this evaluates

Evaluates large language models' ability to answer financial questions using various retrieval and context strategies, probing numerical reasoning, factuality, and handling of structured or long documents.

Datasets

• FinanceBench — total 10231; splits: test (-1); repo https://github.com/patronus-ai/financebench

Metrics

• correct answer (primary) — range: [0, 1]
  • Proportion of model responses labeled as 'correct answer' out of the total evaluated responses. Allows minor unit conversions and rounding errors, but penalizes contradictions with gold evidence or explicit inability to answer.

Input / output format

Input: Question prompt, optionally accompanied by context (none for closed book, gold page text for oracle, retrieved vector store chunks for vector store setups, or full document truncated to ~100k tokens for long context).

Output: Natural language text response answering the financial question.

Scoring recipe

def score_response(response, gold_answer, evidence):
    if 'cannot answer' in response.lower():
        return 'failure_to_answer'
    if matches_gold(response, gold_answer, allow_rounding=True):
        if contradicts_evidence(response, evidence):
            return 'incorrect'
        return 'correct'
    return 'incorrect'

scores = [score_response(r, g, e) for r, g, e in dataset]
accuracy = sum(1 for s in scores if s == 'correct') / len(scores)

Common pitfalls

• Manual labeling was only performed on a 150-question subset, not the full 10,231 benchmark.
• The 'Oracle' setting artificially inflates performance by providing gold evidence, removing the retrieval challenge.
• Long-context setups truncate documents, potentially discarding relevant information and skewing results.
• Allowing minor rounding/unit deviations requires consistent human judgment, which may introduce inter-annotator variance.

Evidence (verbatim from paper)

Models' responses were each assigned to one of three categories. First, correct answer. This is the 'desired' behavior of models. To ensure a good-faith understanding of models' capabilities we allow minor deviations, such as giving the answer in billions when the unit was given in the question as millions. We also allow very small rounding errors. Second, incorrect answer. Incorrect answers vary, from calculations that are off by small margins to several orders of magnitude, and from making up legal information to giving the wrong direction for an effect (e.g. reporting negative growth when it is actually positive). If a model gives the right answer but with logic or calculations that explicitly contradict the evidence in the gold standard answer, we label it Incorrect. Third, failure to answer.

Citation

@misc{islam2023financebench,
  title={FinanceBench: A New Benchmark for Financial Question Answering},
  author={Islam et al. (2023)},
  year={2023},
  note={arXiv:2311.11944}
}

• arXiv: 2311.11944