Fleurs Eval

fleurs-eval

FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech — Conneau et al. (2022) (arXiv:2205.12446, 2022)

What this evaluates

Evaluates universal speech representations across 102 languages using few-shot learning on parallel speech data. Probes capabilities in automatic speech recognition (ASR), speech language identification, and retrieval tasks.

Datasets

• FLEURS — total ?; splits: train (1509), dev (150), test (350)

Metrics

• character level error rate (primary) — range: [0, 1]
  • Standard character-level edit distance normalized by the length of the reference transcript. Calculated on character-based modeling units after NFC/FST normalization, lowercasing, punctuation removal, and word boundary tokenization.

Input / output format

Input: 16kHz audio recordings (≤30 seconds per segment) paired with parallel text transcripts across 102 languages.

Output: Predicted character sequences corresponding to the input audio segments.

Scoring recipe

def calculate_cer(predictions, references):
    total_errors = 0
    total_ref_len = 0
    for pred, ref in zip(predictions, references):
        pred_norm = normalize(pred)
        ref_norm = normalize(ref)
        total_errors += edit_distance(pred_norm, ref_norm)
        total_ref_len += len(ref_norm)
    return total_errors / total_ref_len if total_ref_len > 0 else 0.0

Common pitfalls

• Approximately 21.5% of sentences in the first version lack any validated recordings, requiring careful handling of missing data.
• Speakers are strictly disjoint between train/dev and test sets, so models cannot rely on speaker-specific features.
• Text normalization (NFC, FST, lowercasing, punctuation removal, character splitting) must be applied consistently to both predictions and references to match the stated evaluation unit.

Evidence (verbatim from paper)

Among the various possible modeling units (e.g. character or sentence-pieces) for massively multilingual ASR, a universal vocabulary of characters requires the least resources to build, and better matches a common evaluation metric (i.e. character level error rate).

Citation

@misc{conneau2022fleurs,
  title={FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech},
  author={Conneau et al. (2022)},
  year={2022},
  note={arXiv:2205.12446}
}

• arXiv: 2205.12446