Ccf Aatc 2025 Speech Restoration Eval

ccf-aatc-2025-speech-restoration-eval

The CCF AATC 2025 Speech Restoration Challenge: A Retrospective — Junan Zhang et al. (2025) (arXiv:2509.12974, 2025)

What this evaluates

Evaluates speech restoration models on realistic, multi-stage degradations including acoustic noise/reverberation, codec compression artifacts, and secondary processing artifacts from upstream enhancement models. Probes the trade-off between signal fidelity/intelligibility and perceptual quality while measuring computational efficiency.

Datasets

• CCF AATC 2025 Test Set — total 300; splits: test (300); repo https://github.com/viewfinder-annn/AnyEnhance-v1/releases/download/v0.1_aatc_testset/aatc_testset_v1_release.tar.gz

Metrics

• WAcc (primary) — range: [0, 1]
  • Word Accuracy: percentage of correctly recognized words in the restored speech compared to the clean reference, typically computed via an automatic speech recognition system.
• DNSMOS — range: [0, 5]
  • Deep Noise Suppression Mean Opinion Score, providing three sub-scores: SIG (speech quality), BAK (background noise suppression), and OVRL (overall quality). Computed via a trained neural network on the restored audio.
• PESQ — range: [0, 4.5]
  • Perceptual Evaluation of Speech Quality: standard ITU-T P.862 metric measuring perceived speech quality by aligning and comparing the restored and clean reference waveforms.

Input / output format

Input: Single-channel degraded speech audio utterances (16kHz or similar) mixed with noise, reverberation, MP3 compression, or processed by upstream enhancement models.

Output: Restored speech audio files corresponding to each degraded input utterance.

Scoring recipe

def evaluate_system(restored_audio, clean_audio):
    wacc = asr_accuracy(restored_audio, clean_audio)
    dnsmos = dns_mos_score(restored_audio)  # returns SIG, BAK, OVRL
    pesq_val = pesq_score(restored_audio, clean_audio)
    return wacc, dnsmos, pesq_val

# Aggregate over 300 test utterances (150 acoustic, 50 codec, 100 secondary)
# Compute mean WAcc, mean DNSMOS(SIG/BAK/OVRL), mean PESQ per subset and overall.
# Final ranking score = weighted combination of objective metrics penalized by Params(M).

Common pitfalls

• Generative models often hallucinate or degrade phase coherence, leading to high DNSMOS but low PESQ and WAcc on codec/secondary distortions.
• Subjective MOS evaluation is only conducted on the 150-utterance Acoustic Degradation subset, not the full test set.
• Model complexity (parameter count) is explicitly factored into the final ranking, penalizing large generative models despite high perceptual scores.

Evidence (verbatim from paper)

Metrics include Word Accuracy (WAcc) for speech intelligibility, DNSMOS (SIG, BAK, OVRL) for perceptual quality, and PESQ. “Params” denotes the model parameter count in millions. The best results among the participating teams are highlighted in bold, and the second-best results are underlined.

Citation

@misc{zhang2025ccfaatc,
  title={The CCF AATC 2025 Speech Restoration Challenge: A Retrospective},
  author={Junan Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.12974}
}

• arXiv: 2509.12974