M3finmeeting Eval

m3finmeeting-eval

M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset — Zhu et al. (2025) (arXiv:2506.02510, 2025)

What this evaluates

Probes long-context financial meeting understanding across three languages (EN, ZH, JA) and 11 GICS sectors. It evaluates a model's ability to condense lengthy transcripts into structured summaries, extract relevant question-answer pairs, and localize precise answers within designated sections while ignoring noise.

Datasets

• M3FinMeeting — total 600; splits: test (600); repo https://github.com/aliyun/qwen-dianjin

Metrics

• compression ratio (primary) — range: other
  • Ratio of source document length to reference summary length, computed at both token-level and sentence-level following Koh et al. (2022).

Input / output format

Input: Long-context transcribed financial meeting text (tokenized via tiktoken cl100k_base) in English, Chinese, or Japanese, ranging from ~3.5K to >25K tokens. For QA tasks, context is restricted to a designated section with other sections acting as noise.

Output: Summarization: sequentially concatenated section summaries. QA Pair Extraction: structured pairs of questions and their corresponding answers. QA: direct answer string localized to the designated section.

Scoring recipe

def compute_compression_ratio(source_text, ref_text):
    src_tokens = tokenize(source_text)
    ref_tokens = tokenize(ref_text)
    src_sents = split_sentences(source_text)
    ref_sents = split_sentences(ref_text)
    token_ratio = len(src_tokens) / len(ref_tokens)
    sent_ratio = len(src_sents) / len(ref_sents)
    return token_ratio, sent_ratio

Common pitfalls

• ASR transcripts require manual correction before evaluation; using raw ASR output introduces noise that unfairly penalizes models.
• QA tasks restrict context to a single designated section, with other sections acting as deliberate noise to test precise localization rather than global retrieval.
• Compression ratio is a length-based metric following Koh et al. (2022), not a semantic similarity score like ROUGE or BERTScore.

Evidence (verbatim from paper)

Given transcribed speech documents and their reference summaries, we follow Koh et al. (2022) to compute the compression ratio of a source document length against its reference summary length at both token-level and sentence-level.

Citation

@misc{zhu2025m3finmeeting,
  title={M$^3$FinMeeting: A Multilingual, Multi-Sector, and Multi-Task Financial Meeting Understanding Evaluation Dataset},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2506.02510}
}

• arXiv: 2506.02510