Bizbench Eval

bizbench-eval

BizBench: A Quantitative Reasoning Benchmark for Business and Finance — Koncel-Kedziorski et al. (2023) (arXiv:2311.06602, 2023)

What this evaluates

Evaluates large language models' ability to perform quantitative reasoning in business and finance, specifically focusing on synthesizing executable code from financial questions, extracting numeric values from structured and unstructured data, and applying domain-specific financial knowledge.

Datasets

• BizBench — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total predictions) * 100.

Input / output format

Input: Financial questions requiring quantitative reasoning, optionally accompanied by few-shot demonstrations in the context window.

Output: Executable code to solve the problem, or direct text/numeric answers depending on the specific task (e.g., FinCode, SEC-Num, FormulaEval).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if pred == gold_item: # or code execution matches expected output
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Performance varies drastically across sub-tasks (e.g., FinCode is significantly harder than others), so reporting a single aggregate score without task breakdown can be misleading.
• Model size and alignment methods (RLHF, instruction tuning) heavily influence results, making comparisons between base and instruction-tuned models difficult without careful normalization.
• Even state-of-the-art models fail on a large portion of questions (e.g., GPT-4 fails 36.4%), so accuracy alone may mask partial credit or code execution nuances.

Evidence (verbatim from paper)

Significantly, when Llama-2-7B is finetuned with the entire training dataset for each task, it demonstrates a substantial improvement in performance – achieving a 9%, 7%, and 13% higher accuracy on CodeFinQA, CodeTAT-QA, and SEC-Num, respectively compared to Llama-2-70B – while incurring only a fraction of the inference cost.

Citation

@misc{koncelkedziorski2023bizbench,
  title={BizBench: A Quantitative Reasoning Benchmark for Business and Finance},
  author={Koncel-Kedziorski et al. (2023)},
  year={2023},
  note={arXiv:2311.06602}
}

• arXiv: 2311.06602