qhjqhj00 fleurs-slu-eval
Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding — Schmidt et al. (2025) (arXiv:2501.06117, 2025)
What this evaluates
Evaluates multilingual spoken language understanding (SLU) across 102 languages for topical classification and 92 languages for spoken multiple-choice QA, testing cross-lingual transfer, speech-to-text translation, and robustness to audio quality variations.
Datasets
- SIB-Fleurs — total ?; splits: train (-1), test (-1)
- Belebele-Fleurs — total ?; splits: train (-1), dev (-1), test (-1)
Metrics
accuracy(primary) — range: [0, 1]- Standard classification and multiple-choice QA accuracy: the fraction of instances where the predicted label or choice matches the gold label.
cross-entropy— range: [0, inf)- Cross-entropy loss computed over concatenated choice logits to evaluate model confidence and training convergence on the QA task.
CER— range: [0, 1]- Character Error Rate computed between reference Flores sentences and ASR transcriptions to quantify utterance quality for subset analysis.
Input / output format
Input: For classification: speech utterances or transcriptions. For QA: paragraph, question, and multiple-choice options in text or speech modality. Speech-LLMs receive an English task description plus in-language input.
Output: Predicted class label or correct choice index.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels)
def compute_cer(reference, hypothesis):
# Standard edit distance based CER
return edit_distance(reference, hypothesis) / len(reference)
Common pitfalls
- Audio must be normalized to an RMS level of 0.07 to ensure consistent quality across languages.
- For languages unsupported by the ASR backends, the closest available language must be manually selected for in-language transcription.
- Speech LLMs based on Whisper-v3 are limited to 30 seconds of audio, so longer Belebele-Fleurs instances cannot be evaluated with them.
Evidence (verbatim from paper)
We train and validate CS models on the English training and dev sets of Belebele (Bandarkar et al., 2024), respectively. We jointly embed the paragraph, question, and choices with text encoders. We then average the token encodings of each choice $c_{i}\in C$ and project the choice embedding via head $H^{D\times 1}$ to a logit $\mathbf{l_{c_{i}}}$. We minimize the cross-entropy of the concatenated choice logits ${\mathbf{l_{c_{i}}}}_{i=1}^{|C|}$ to the label choice.
Citation
@misc{schmidt2025fleurs_slu,
title={Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding},
author={Schmidt et al. (2025)},
year={2025},
note={arXiv:2501.06117}
}
- arXiv: 2501.06117