Mgb3 Eval

mgb3-eval

Speech Recognition Challenge in the Wild: Arabic MGB-3 — Ali et al. (2017) (arXiv:1709.07276, 2017)

What this evaluates

Evaluates automatic speech recognition and Arabic dialect identification in uncontrolled, real-world settings with high dialectal and genre diversity. It probes robustness to orthographic variability and low-resource conditions by measuring transcription accuracy against multiple human references.

Datasets

• MGB-3 — total ?; splits: dev (1279)

Metrics

• MR-WER (primary) — range: percent
  • Multi-Reference Word Error Rate. Computes standard WER against each available manual transcription and returns the minimum value. A recognized word is accepted if it appears in any reference at the same position.
• WER — range: percent
  • Standard Word Error Rate measuring the percentage of insertions, deletions, and substitutions relative to a single reference transcription.
• Accuracy — range: percent
  • Percentage of correctly predicted dialect labels in the Arabic Dialect Identification (ADI) task across five classes.

Input / output format

Input: Raw audio speech signal (baseline systems may also use extracted lexical or i-vector features).

Output: For ASR: a transcribed text string. For ADI: a single dialect label from {Egyptian, Levantine, Gulf, North African, MSA}.

Scoring recipe

def compute_mr_wer(hypothesis, references):
    wers = []
    for ref in references:
        wers.append(compute_standard_wer(hypothesis, ref))
    return min(wers)

# For ADI task:
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Inter-annotator disagreement is high (~13%) due to the lack of standardized orthography in dialectal Arabic, making single-reference WER misleading.
• Systems trained on MGB-2 perform poorly on MGB-3 without adaptation to the changed genre and dialect characteristics.
• Surface text normalization must be applied before evaluation to account for orthographic variability across annotators.

Evidence (verbatim from paper)

For the MGB-3 Challenge, we investigated using MR-WER. This metric is based on comparing the recognized text against multiple manual transcriptions of the speech signal, which are all considered valid references. This approach thus accepts a recognized word if any of the references include it in the same form.

Citation

@misc{ali2017mgb3,
  title={Speech Recognition Challenge in the Wild: Arabic MGB-3},
  author={Ali et al. (2017)},
  year={2017},
  note={arXiv:1709.07276}
}

• arXiv: 1709.07276