Ucfe Eval

ucfe-eval

UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models — Yang et al. (2024) (arXiv:2410.14059, 2024)

What this evaluates

Evaluates large language models' ability to handle dynamic, multi-turn financial dialogues across diverse user personas and task types, measuring their adaptability to shifting user needs and financial expertise.

Datasets

• UCFE — total 330; splits: test (330); repo https://github.com/TobyYang7/UCFE-Benchmark

Metrics

• Elo score (primary) — range: other
  • Pairwise comparison rating system where models earn points based on win/loss/draw outcomes against other models in head-to-head dialogues.
• Pearson correlation — range: [-1, 1]
  • Measures linear correlation between model Elo scores and human expert preference ratings to validate alignment.

Input / output format

Input: Multi-turn financial dialogues categorized by 4 user types (analysts, professionals, regulators, public) and 17 task types, provided in zero-shot or few-shot prompt formats.

Output: Model-generated responses per dialogue turn, evaluated via pairwise comparison and LLM-as-Judge scoring.

Scoring recipe

# 1. Generate model responses for each multi-turn dialogue
predictions = model.generate(dialogue_history)
# 2. Pairwise comparison via LLM-as-Judge or human experts
win_rate = count_wins(predictions) / total_comparisons
# 3. Compute Elo rating from win rates
elo = 400 * log10(win_rate / (1 - win_rate))
# 4. Validate against human preferences
pearson_r = np.corrcoef(elo_scores, human_scores)[0, 1]

Common pitfalls

• Zero-shot task text limitations can artificially depress model performance scores.
• Dynamic task evolution requires models to adapt mid-dialogue, making static evaluation metrics insufficient.
• LLM-as-Judge scoring may inherit biases from the judge model's training data.

Evidence (verbatim from paper)

Table D-1 presents the Elo scores across all tasks. The results show that LLMs trained on financial text data demonstrate outstanding performance across various tasks. However, due to limitations in the zero-shot task texts, the models' performance in those tasks may be affected.

Citation

@misc{yang2024ucfe,
  title={UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2410.14059}
}

• arXiv: 2410.14059