Bankertoolbench Eval

bankertoolbench-eval

BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows — Lau et al. (2026) (arXiv:2604.11304, 2026)

What this evaluates

Evaluates AI agents' ability to execute end-to-end investment banking workflows, requiring multi-file deliverable generation (Excel, PowerPoint, reports) using specialized financial tools and data sources. It probes financial judgment, cross-artifact consistency, tool-use fidelity, and professional presentation standards under realistic constraints.

Datasets

• BankerToolBench — total 100; splits: train (80), test (20); repo https://github.com/harbor-framework/harbor

Metrics

• rubric score (primary) — range: [0, 1]
  • Average score across 100+ expert-crafted criteria per task. Each criterion is evaluated as a binary pass/fail based on professional deliverable standards, then averaged to produce a single task-level metric.

Input / output format

Input: A structured prompt template specifying the agent's role ('investment banker'), available tools (MCP tools for VDR/SEC EDGAR, Python libraries), directory layout, execution protocol, formatting guidelines, security constraints, and a specific instruction.

Output: Multi-file deliverables (e.g., .xlsx, .pptx, .pdf, .docx) saved to the banker Workspace/deliverables/ directory. Intermediate scripts go to banker Workspace/.

Scoring recipe

def compute_rubric_score(predictions, rubric_criteria):
    scores = []
    for criterion in rubric_criteria:
        # Binary check: 1 if deliverable meets criterion, 0 otherwise
        met = verifier_or_human_check(predictions, criterion)
        scores.append(1.0 if met else 0.0)
    return sum(scores) / len(scores)

Common pitfalls

• Excel formulas written by agents must be recalculated using headless LibreOffice before being read with openpyxl, otherwise they return None or stale values.
• Agents must strictly use the provided MCP tools (VDR, SEC EDGAR) for financial data and are explicitly forbidden from using web search or external APIs.
• Only files saved to banker Workspace/deliverables/ are reviewed for grading; intermediate outputs in other directories are ignored.

Evidence (verbatim from paper)

Table 4 shows that for Claude Opus 4.6, the spread in output quality achieved across the three harnesses is very small (the average rubric score only varies by 0.3).

Citation

@misc{lau2026bankertoolbench,
  title={BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows},
  author={Lau et al. (2026)},
  year={2026},
  note={arXiv:2604.11304}
}

• arXiv: 2604.11304