Flame Cer Eval

flame-cer-eval

FLAME: Financial Large-Language Model Assessment and Metrics Evaluation — Guo et al. (2025) (arXiv:2501.06211, 2025)

What this evaluates

Evaluates financial domain knowledge and certification exam readiness in Chinese and English. It probes models' ability to answer multiple-choice questions across 14 professional financial certifications with varying difficulty levels.

Datasets

• FLAME-Cer — total 16000; splits: test (-1); repo https://github.com/FLAME-ruc/FLAME

Metrics

• accuracy rate (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100.

Input / output format

Input: Multiple-choice question text in Chinese or English, followed by options (A, B, C, D, or E).

Output: A single letter corresponding to the selected option (e.g., 'A', 'B', 'C', 'D').

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(predictions)) * 100
return accuracy

Common pitfalls

• Models may struggle with bilingual prompts (Chinese questions with English options or vice versa).
• Strict format adherence is required; outputting explanations alongside the letter may cause parsing failures.
• Financial calculations and regulatory details require precise domain knowledge; hallucination rates are high for complex actuarial or accounting questions.

Evidence (verbatim from paper)

The performance comparison of LLMs on the FLAME-Cer benchmark is shown in Table[1]. We could find that Baichuan4-Finance leads with an average accuracy rate of 93.62%, followed by Qwen2.5-72B-Instruct at 88.24%.

Citation

@misc{guo2025flame,
  title={FLAME: Financial Large-Language Model Assessment and Metrics Evaluation},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2501.06211}
}

• arXiv: 2501.06211