Atc Asr Csd Eval

atc-asr-csd-eval

The Airbus Air Traffic Control speech recognition 2018 challenge: towards ATC automatic transcription and call sign detection — Pellegrini et al. (2018) (arXiv:1810.12614, 2018)

What this evaluates

Evaluates automatic speech recognition (ASR) and call sign detection (CSD) capabilities on real-world air traffic control audio. It probes the model's ability to transcribe accented, noisy pilot and controller speech and accurately identify aircraft call signs in domain-specific phraseology.

Datasets

• Airbus ATC Speech Recognition 2018 Challenge Corpus — total ?; splits: eval (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=number of reference words.
• CSD F1-score — range: percent
  • F1-score for Call Sign Detection: harmonic mean of precision and recall computed over detected call sign spans.

Input / output format

Input: Audio recordings of air traffic control communications (pilot and controller speech) with varying noise levels and accents.

Output: For ASR: word-level transcription. For CSD: detected call sign spans or labels.

Scoring recipe

# ASR WER
wer = (substitutions + deletions + insertions) / len(reference_words)

# CSD F1
precision = true_positives / (true_positives + false_positives)
recall = true_positives / (true_positives + false_negatives)
f1 = 2 * (precision * recall) / (precision + recall)

Common pitfalls

• Call signs are often shortened or used in partial forms, making detection harder than strict rule-based matching.
• Pilots' speech has significantly higher WER than controllers' due to greater accent diversity and faster speech rates.
• Some teams submitted only ASR results without CSD predictions, resulting in zero CSD scores that skew overall rankings.

Evidence (verbatim from paper)

Figures [1]a and [1]b show the Word Error Rates (WER) for the ASR task and the F1-scores for CSD, obtained by the 22 teams ordered by their final ranking. VOCAPIA-LIMSI achieved the best results in both tasks with a 7.62% WER and a 82.41% CSD F1-score.

Citation

@misc{pellegrini2018airbus,
  title={The Airbus Air Traffic Control speech recognition 2018 challenge: towards ATC automatic transcription and call sign detection},
  author={Pellegrini et al. (2018)},
  year={2018},
  note={arXiv:1810.12614}
}

• arXiv: 1810.12614