# Flame Sce Eval

> Assesses practical financial application capabilities across a hierarchical framework of 10 primary and 21 secondary scenarios. It probes models' ability to perform real-world financial tasks such as compliance checking, document generation, risk control, data extraction, and client analysis. Use when the user wants to benchmark on FLAME-Sce, or asks about evaluating this task. Reports usability rate.

- Skill: `qhjqhj00/flame-sce-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/flame-sce-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/flame-sce-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/flame-sce-eval

---


# flame-sce-eval

> FLAME: Financial Large-Language Model Assessment and Metrics Evaluation — Guo et al. (2025) (arXiv:2501.06211, 2025)

## What this evaluates

Assesses practical financial application capabilities across a hierarchical framework of 10 primary and 21 secondary scenarios. It probes models' ability to perform real-world financial tasks such as compliance checking, document generation, risk control, data extraction, and client analysis.

## Datasets

- **FLAME-Sce** — total ?; splits: test (-1); repo https://github.com/FLAME-ruc/FLAME

## Metrics

- `usability rate` **(primary)** — range: percent
  - Percentage of tasks completed successfully according to predefined format and content rules. Calculated as (number of successfully evaluated tasks / total tasks) * 100.

## Input / output format

**Input**: Task-specific prompt containing instructions, context (e.g., financial reports, customer profiles, transaction records), and strict output formatting requirements.

**Output**: Varies by scenario: structured text, JSON, classification labels, or specific formatted responses (e.g., '我的结论是：[result]').

## Scoring recipe

```python
success_count = 0
for pred, gold in zip(predictions, gold_labels):
    if check_format(pred, gold['format']) and check_content(pred, gold['answer']):
        success_count += 1
usability_rate = (success_count / len(predictions)) * 100
return usability_rate
```

## Common pitfalls

- Strict output formatting is enforced; deviating from the requested structure (e.g., JSON, specific prefixes) results in automatic failure.
- Open-ended scenarios (e.g., marketing, analysis) require balancing creativity with strict financial compliance and length constraints.
- Models often fail to correctly parse or extract data from OCR/text-heavy financial documents.

## Evidence (verbatim from paper)

> Overall, Baichuan4-Finance leads with a usability rate of 84.15%. GPT-4o and Qwen2.5-72B-Instruct exhibit similar performance levels, around 79%.

## Citation

```bibtex
@misc{guo2025flame,
  title={FLAME: Financial Large-Language Model Assessment and Metrics Evaluation},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2501.06211}
}
```

- arXiv: 2501.06211