Echox Speech QA Eval

echox-speech-qa-eval

EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs — Zhang et al. (2025) (arXiv:2509.09174, 2025)

What this evaluates

Evaluates the knowledge-based question-answering capabilities of speech-to-speech and speech-to-text models on audio and text inputs.

Datasets

• Llama Questions — total ?; splits: test (-1)
• Web Questions — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 100] percent
  • Exact match or F1 score computed by the UltraEval-Audio toolkit on knowledge-based QA benchmarks. Values are reported as percentages averaged across datasets.

Input / output format

Input: Spoken audio questions (for speech-to-speech evaluation) or transcribed text questions (for speech-to-text evaluation).

Output: Generated spoken audio responses (for speech-to-speech) or text responses (for speech-to-text), evaluated against gold answers.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    scores = []
    for pred, gold in zip(predictions, gold_answers):
        # UltraEval-Audio handles transcription if needed, then computes exact match/F1
        score = ultra_eval_toolkit.compute_qa_score(pred, gold)
        scores.append(score)
    return sum(scores) / len(scores) * 100

Common pitfalls

• The evaluation relies on the UltraEval-Audio toolkit; using a different toolkit may yield different scores due to variations in ASR transcription or QA scoring logic.
• Speech-to-speech results depend on the unit-based HiFi-GAN vocoder for audio generation, but the reported metric focuses on QA accuracy rather than audio quality.
• The paper reports an 'Avg.' column across the three benchmarks, which should be computed as the mean of the individual dataset scores.

Evidence (verbatim from paper)

For evaluation, we use the UltraEval-Audio toolkit. We mainly conduct experiments on the three benchmarks: Llama questions (Nachmani et al., 2023), Web questions (Berant et al., 2013), and TriviaQA (Joshi et al., 2017).

Citation

@misc{zhang2025echox,
  title={EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.09174}
}

• arXiv: 2509.09174