Dowis Eval

dowis-eval

Do What I Say: A Spoken Prompt Dataset for Instruction-Following — Züfle et al. (2026) (arXiv:2603.09881, 2026)

What this evaluates

This benchmark evaluates instruction-following capabilities of speech-large language models (SLLMs) by comparing performance when prompted with text versus spoken audio across nine diverse tasks. It probes cross-lingual generalization, prompt style robustness, and the model's ability to handle both text and speech modalities for input and output.

Datasets

• DOWIS (Do What I Say) — total ?; splits: test (-1); repo https://github.com/MaikeZuefle/DOWIS
• FLEURS — total ?; splits: test (-1)
• MCIF — total ?; splits: test (-1)
• YTSeg — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate measures the percentage of words incorrectly recognized or generated compared to a reference transcript, calculated as (insertions + deletions + substitutions) / total words.
• CometKiwi — range: [0, 1]
  • A neural quality estimation metric that predicts translation quality without requiring reference translations, scoring based on contextual embeddings of source and hypothesis.
• BERTScore — range: [0, 1]
  • Computes token-level semantic similarity between generated and reference texts using a pre-trained DeBERTa model, normalized to [0, 1].
• UTMOS — range: [1, 5]
  • A deep learning-based metric that predicts the mean opinion score (MOS) of speech quality on a scale from 1 to 5.
• Collar-F1 — range: [0, 1]
  • A chunk-based evaluation metric for audio segmentation that measures the F1 score of predicted vs. reference segments within a ±3-second tolerance window.
• BERTScore_GC — range: [0, 1]
  • BERTScore computed on globally concatenated predicted and reference titles to assess overall semantic alignment.

Input / output format

Input: Text or spoken audio prompts in the target language, formatted in one of five styles (basic, formal, informal, short, detailed).

Output: Text responses for instruction-following tasks (ASR, SQA, TSUM, SSUM, ACHAP, MT, ST) or audio responses for generation tasks (TTS, S2ST).

Scoring recipe

# ASR & TTS
wer = jiwer.compute_wer(predictions, references)
# MT, ST, S2ST (content accuracy)
comet_score = cometkiwi.predict(predictions, references)
# SQA, TSUM, SSUM
bert_score = bertscore.compute(predictions, references, model="microsoft/deberta-xlarge-mnli", normalize=True)
# ACHAP
collar_f1 = chunkseg.evaluate(predictions, references, window=3.0)
bertscore_gc = bertscore.compute(concat(predictions), concat(references))

Common pitfalls

• Text prompts consistently outperform spoken prompts in text-output tasks, particularly in low-resource and cross-lingual settings, which can mask speech-specific degradation.
• Speech-output tasks (TTS, S2ST) are only evaluated on Qwen2.5-Omni because Phi-4-multimodal-instruct lacks audio generation capabilities, limiting cross-model comparison for these tasks.
• Cross-lingual tasks require prompts to be formulated in the target language, not the source language, which differs from standard machine translation setups.

Evidence (verbatim from paper)

We evaluate all tasks using standard metrics on the given datasets. Specifically, we evaluate ASR using Word Error Rate (WER) computed with jiwer. For MT and ST, we use CometKiwi, a quality estimation metric that does not require reference translations that has shown good correlation with humans for speech and text translation. For SQA, SSUM, and TSUM, we follow the evaluation protocol from MCIF, using normalized BERTScore with the deberta-xlarge-mnli model to measure semantic similarity between generated and reference answers. For speech-output tasks (TTS and S2ST), we first transcribe the generated audio using whisper-large-v3. We then report WER for TTS and CometKiwi for S2ST to evaluate content accuracy. To assess speech quality, we additionally report UTMOS for both tasks.

Citation

@misc{zuefle2026dowis,
  title={Do What I Say: A Spoken Prompt Dataset for Instruction-Following},
  author={Züfle et al. (2026)},
  year={2026},
  note={arXiv:2603.09881}
}

• arXiv: 2603.09881