# Ursa Gan Eval

> Evaluates cross-domain speech recognition and enhancement robustness by training downstream models on generatively simulated target-domain data. Probes the ability of ASR and SE systems to generalize to unseen acoustic conditions, channel mismatches, and compound noise-channel distortions. Use when the user wants to benchmark on Hakka Across Taiwan (HAT), Taiwanese Across Taiwan (TAT), VoiceBank-DEMAND (VBD), HAT-ESC, or asks about evaluating this task. Reports CER.

- Skill: `qhjqhj00/ursa-gan-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/ursa-gan-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ursa-gan-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/ursa-gan-eval

---


# ursa-gan-eval

> Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement — Wang et al. (2026) (arXiv:2602.04307, 2026)

## What this evaluates

Evaluates cross-domain speech recognition and enhancement robustness by training downstream models on generatively simulated target-domain data. Probes the ability of ASR and SE systems to generalize to unseen acoustic conditions, channel mismatches, and compound noise-channel distortions.

## Datasets

- **Hakka Across Taiwan (HAT)** — total 815552; splits: train (779080), test (36472)
- **Taiwanese Across Taiwan (TAT)** — total ?; splits: train (-1), test (-1)
- **VoiceBank-DEMAND (VBD)** — total 12356; splits: train (11572), test (784)
- **HAT-ESC** — total ?; splits: train (-1), test (-1)

## Metrics

- `CER` **(primary)** — range: [0, 1]
  - Character Error Rate: ratio of character-level insertions, deletions, and substitutions to the total number of characters in the reference transcription.
- `PESQ` — range: other
  - Perceptual Evaluation of Speech Quality: compares enhanced speech to a clean reference signal to estimate perceptual quality, correlating with human judgments.
- `STOI` — range: [0, 1]
  - Short-Time Objective Intelligibility: measures temporal and spectral similarity between enhanced and reference signals to assess speech intelligibility under noise.
- `MOS` — range: [1, 5]
  - Mean Opinion Score: average rating provided by human listeners to evaluate the naturalness and authenticity of synthesized speech.

## Input / output format

**Input**: Source-domain speech utterances (spectrograms segmented into 129x128 patches) and 40 target-domain utterances for GAN training; downstream models receive simulated target-domain speech and original source speech for training, and held-out target-domain test utterances for evaluation.

**Output**: ASR: character-level transcriptions. SE: enhanced waveform. Simulated data: generated speech aligned with target conditions.

## Scoring recipe

```python
def compute_metrics(predictions, gold, clean_ref=None, listener_ratings=None):
    # ASR
    cer = edit_distance(predictions['asr_chars'], gold['chars']) / len(gold['chars'])
    # SE
    pesq = compute_pesq(clean_ref, predictions['se_waveform'])
    stoi = compute_stoi(clean_ref, predictions['se_waveform'])
    # MOS
    mos = sum(listener_ratings) / len(listener_ratings) if listener_ratings else None
    return {'CER': cer, 'PESQ': pesq, 'STOI': stoi, 'MOS': mos}
```

## Common pitfalls

- GAN training uses only 40 target utterances, which may not represent the full target domain distribution and could inflate generalization claims.
- VBD test set has 40 utterances removed for GAN training, altering the standard evaluation split and reducing test size from 824 to 784.
- Metrics are computed on downstream models (WhisperTiny, DEMUCS) rather than directly on URSA-GAN, so reported performance depends heavily on the downstream architecture choice.

## Evidence (verbatim from paper)

> For ASR, the character error rate was used, which calculates the ratio of character-level insertions, deletions, and substitutions to the total number of characters, providing an accurate transcription assessment, especially for languages with complex orthographies. For SE, we employed the perceptual evaluation of speech quality (PESQ) *[[46]-a new method for speech quality assessment of telephone networks and codecs")]* and the short-time objective intelligibility (STOI) *[[49]]*. PESQ estimates the perceptual quality of enhanced speech by comparing it to a clean reference, correlating well with human judgments. STOI measures speech intelligibility by analyzing temporal and spectral similarity between the enhanced and reference signals, effectively assessing intelligibility under various noise conditions.

## Citation

```bibtex
@misc{wang2026ursa,
  title={Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.04307}
}
```

- arXiv: 2602.04307