# M3finmeeting Eval

> Probes long-context financial meeting understanding across three languages (EN, ZH, JA) and 11 GICS sectors. It evaluates a model's ability to condense lengthy transcripts into structured summaries, extract relevant question-answer pairs, and localize precise answers within designated sections while ignoring noise. Use when the user wants to benchmark on M3FinMeeting, or asks about evaluating this task. Reports compression ratio.

- Skill: `qhjqhj00/m3finmeeting-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/m3finmeeting-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/m3finmeeting-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/m3finmeeting-eval

---


# m3finmeeting-eval

> M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset — Zhu et al. (2025) (arXiv:2506.02510, 2025)

## What this evaluates

Probes long-context financial meeting understanding across three languages (EN, ZH, JA) and 11 GICS sectors. It evaluates a model's ability to condense lengthy transcripts into structured summaries, extract relevant question-answer pairs, and localize precise answers within designated sections while ignoring noise.

## Datasets

- **M3FinMeeting** — total 600; splits: test (600); repo https://github.com/aliyun/qwen-dianjin

## Metrics

- `compression ratio` **(primary)** — range: other
  - Ratio of source document length to reference summary length, computed at both token-level and sentence-level following Koh et al. (2022).

## Input / output format

**Input**: Long-context transcribed financial meeting text (tokenized via tiktoken cl100k_base) in English, Chinese, or Japanese, ranging from ~3.5K to >25K tokens. For QA tasks, context is restricted to a designated section with other sections acting as noise.

**Output**: Summarization: sequentially concatenated section summaries. QA Pair Extraction: structured pairs of questions and their corresponding answers. QA: direct answer string localized to the designated section.

## Scoring recipe

```python
def compute_compression_ratio(source_text, ref_text):
    src_tokens = tokenize(source_text)
    ref_tokens = tokenize(ref_text)
    src_sents = split_sentences(source_text)
    ref_sents = split_sentences(ref_text)
    token_ratio = len(src_tokens) / len(ref_tokens)
    sent_ratio = len(src_sents) / len(ref_sents)
    return token_ratio, sent_ratio
```

## Common pitfalls

- ASR transcripts require manual correction before evaluation; using raw ASR output introduces noise that unfairly penalizes models.
- QA tasks restrict context to a single designated section, with other sections acting as deliberate noise to test precise localization rather than global retrieval.
- Compression ratio is a length-based metric following Koh et al. (2022), not a semantic similarity score like ROUGE or BERTScore.

## Evidence (verbatim from paper)

> Given transcribed speech documents and their reference summaries, we follow Koh et al. (2022) to compute the compression ratio of a source document length against its reference summary length at both token-level and sentence-level.

## Citation

```bibtex
@misc{zhu2025m3finmeeting,
  title={M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2506.02510}
}
```

- arXiv: 2506.02510