Finteam Financial Eval

finteam-financial-eval

FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios — Wu et al. (2025) (arXiv:2507.10448, 2025)

What this evaluates

Evaluates a multi-agent financial system's ability to answer real-world investor inquiries across macroeconomic, industry, and company analysis scenarios. Probes accuracy, thoroughness, clarity, and professional financial reasoning through automated LLM-judging and human preference testing.

Datasets

• NGA Grand Era Investor Inquiries — total 150; splits: test (150); repo https://github.com/FudanDISC/DISC-FinLLM

Metrics

• Overall Score (primary) — range: [1, 5]
  • Average of 1-to-5 ratings assigned by GPT-4o across four dimensions: Accuracy, Thoroughness, Clarity, and Professionalism.
• Acceptance Rate — range: percent
  • Percentage of test cases where the model's response is selected as the best among all compared models by human evaluators.

Input / output format

Input: Natural language investor inquiries regarding macroeconomic trends, industry sub-sectors, or publicly listed companies.

Output: Natural language financial analysis/response addressing the inquiry.

Scoring recipe

# GPT-4o Automated Scoring
gpt_scores = []
for response in responses:
    score = gpt4o_judge.evaluate(response, criteria=["Accuracy", "Thoroughness", "Clarity", "Professionalism"])
    gpt_scores.append(score.overall)  # 1-5 scale
overall_score = mean(gpt_scores)

# Human Acceptance Rate
human_picks = 0
for case in test_cases:
    best_model = human_evaluator.select_best([model_outputs[case]])
    if best_model == target_model:
        human_picks += 1
acceptance_rate = (human_picks / len(test_cases)) * 100

Common pitfalls

• GPT-4o is used as an automated judge, which may introduce LLM-judge bias or over-reward verbose but shallow responses.
• Human evaluators are finance undergraduates rather than professional analysts, potentially affecting the reliability of the Acceptance Rate.
• Acceptance Rate is calculated over a multi-way comparison (5 models) rather than pairwise, making direct comparison with pairwise benchmarks difficult.

Evidence (verbatim from paper)

For evaluation, we employ GPT-4o to score the outputs from our agent system and the other models, ensuring objectivity and accuracy. The evaluation is conducted across four dimensions: (1) Accuracy: The model addresses key points directly, avoiding irrelevant details. (2) Thoroughness: The model provides a detailed, in-depth answer. (3) Clarity: The response is clear, concise, and logical. (4) Professionalism: The model uses appropriate financial perfessional terms. We compare model outputs across multiple dimensions, with GPT-4o rating each response from 1 to 5 per category and assigning an overall score. ... The Acceptance Rate indicates how often a model's output is chosen as the top answer.

Citation

@misc{wu2025finteam,
  title={FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2507.10448}
}

• arXiv: 2507.10448