Taigispeech Eval

taigispeech-eval

TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild — Kai-Wei Chang et al. (arXiv:2603.21478, 2026)

What this evaluates

Evaluates speech intent recognition in a low-resource, real-world setting for Taiwanese Taigi, specifically testing domain adaptation and robustness to domain mismatch between mined training data and real-world elderly speech.

Datasets

• TaigiSpeech — total 3079; splits: test (960), fine-tune (1600)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy (correct predictions / total predictions). The exact metric name is not explicitly stated in the provided section, but accuracy is standard for the described classification tasks.

Input / output format

Input: Raw audio utterances (and video clips for the audio-visual mining baseline setup).

Output: Discrete class label: 5-class (four emergency intents + Non-Emergency) or binary (Emergency vs. Non-Emergency).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Domain mismatch: Training data comes from mined drama/video clips, while test data is real-world elderly speech, potentially causing performance drops not captured by in-distribution metrics.
• Pseudo-labels treated as ground truth: The drama dataset evaluation uses Gemini-3 pseudo-labels as ground truth, which may overestimate performance if pseudo-labels contain errors.
• Class distribution: The test set is balanced by speaker selection, but real-world deployment may see different intent frequencies.

Evidence (verbatim from paper)

To construct a balanced evaluation benchmark, we select 6 speakers from TaigiSpeech to form the test set. The set consists of 3 male and 3 female speakers, with ages distributed to maintain balance across gender and age groups. Each speaker contributes 160 utterances, resulting in a total of 960 test samples. ... Models trained on the mined data are evaluated on the drama dataset (denoted as Drama222Pseudo labels obtained from Gemini-3 are treated as ground truth) and the real-world recording dataset TaigiSpeech (denoted as Taigi).

Citation

@misc{chang2026taigispeech,
  title={TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild},
  author={Kai-Wei Chang et al.},
  year={2026},
  note={arXiv:2603.21478}
}

• arXiv: 2603.21478