Ursa Gan Eval

ursa-gan-eval

Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement — Wang et al. (2026) (arXiv:2602.04307, 2026)

What this evaluates

Evaluates cross-domain speech recognition and enhancement robustness by training downstream models on generatively simulated target-domain data. Probes the ability of ASR and SE systems to generalize to unseen acoustic conditions, channel mismatches, and compound noise-channel distortions.

Datasets

• Hakka Across Taiwan (HAT) — total 815552; splits: train (779080), test (36472)
• Taiwanese Across Taiwan (TAT) — total ?; splits: train (-1), test (-1)
• VoiceBank-DEMAND (VBD) — total 12356; splits: train (11572), test (784)
• HAT-ESC — total ?; splits: train (-1), test (-1)

Metrics

• CER (primary) — range: [0, 1]
  • Character Error Rate: ratio of character-level insertions, deletions, and substitutions to the total number of characters in the reference transcription.
• PESQ — range: other
  • Perceptual Evaluation of Speech Quality: compares enhanced speech to a clean reference signal to estimate perceptual quality, correlating with human judgments.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility: measures temporal and spectral similarity between enhanced and reference signals to assess speech intelligibility under noise.
• MOS — range: [1, 5]
  • Mean Opinion Score: average rating provided by human listeners to evaluate the naturalness and authenticity of synthesized speech.

Input / output format

Input: Source-domain speech utterances (spectrograms segmented into 129x128 patches) and 40 target-domain utterances for GAN training; downstream models receive simulated target-domain speech and original source speech for training, and held-out target-domain test utterances for evaluation.

Output: ASR: character-level transcriptions. SE: enhanced waveform. Simulated data: generated speech aligned with target conditions.

Scoring recipe

def compute_metrics(predictions, gold, clean_ref=None, listener_ratings=None):
    # ASR
    cer = edit_distance(predictions['asr_chars'], gold['chars']) / len(gold['chars'])
    # SE
    pesq = compute_pesq(clean_ref, predictions['se_waveform'])
    stoi = compute_stoi(clean_ref, predictions['se_waveform'])
    # MOS
    mos = sum(listener_ratings) / len(listener_ratings) if listener_ratings else None
    return {'CER': cer, 'PESQ': pesq, 'STOI': stoi, 'MOS': mos}

Common pitfalls

• GAN training uses only 40 target utterances, which may not represent the full target domain distribution and could inflate generalization claims.
• VBD test set has 40 utterances removed for GAN training, altering the standard evaluation split and reducing test size from 824 to 784.
• Metrics are computed on downstream models (WhisperTiny, DEMUCS) rather than directly on URSA-GAN, so reported performance depends heavily on the downstream architecture choice.

Evidence (verbatim from paper)

For ASR, the character error rate was used, which calculates the ratio of character-level insertions, deletions, and substitutions to the total number of characters, providing an accurate transcription assessment, especially for languages with complex orthographies. For SE, we employed the perceptual evaluation of speech quality (PESQ) [[46]-a new method for speech quality assessment of telephone networks and codecs")] and the short-time objective intelligibility (STOI) [[49]]. PESQ estimates the perceptual quality of enhanced speech by comparing it to a clean reference, correlating well with human judgments. STOI measures speech intelligibility by analyzing temporal and spectral similarity between the enhanced and reference signals, effectively assessing intelligibility under various noise conditions.

Citation

@misc{wang2026ursa,
  title={Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.04307}
}

• arXiv: 2602.04307