Svqa Vqa Eval

svqa-vqa-eval

Spoken question answering for visual queries — Shabtay et al. (2025) (arXiv:2505.23308, 2025)

What this evaluates

Evaluates a multimodal model's ability to answer visual questions when the query is provided as spoken audio rather than text. It probes speech-vision-language alignment, robustness to synthesized speech variations, and the model's capacity to handle modality-specific prompts.

Datasets

• SEED-Bench — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• DocVQA — total ?; splits: val (-1)
• MLS — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of predicted answers that exactly match the ground-truth answer (case-insensitive).
• ANLS — range: [0, 1]
  • Average Normalized Levenshtein Similarity; computes the normalized edit distance between predicted and ground-truth answers, averaged across the dataset.
• WER — range: percent
  • Word Error Rate; percentage of words incorrectly transcribed relative to the reference transcript.

Input / output format

Input: Image paired with spoken audio containing a question (synthesized via StyleTTS2 or F5-TTS), optionally accompanied by a textual prompt.

Output: Textual answer to the visual question, or a transcript of the input speech for WER evaluation.

Scoring recipe

def compute_metrics(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower()) / len(golds)
    elif metric_type == 'ANLS':
        return sum(normalized_levenshtein(p, g) for p, g in zip(predictions, golds)) / len(golds)
    elif metric_type == 'WER':
        return sum(word_error_rate(p, g) for p, g in zip(predictions, golds)) / len(golds) * 100

Common pitfalls

• Model may ignore speech instructions and treat audio content as a VQA question instead of transcribing it, leading to artificially high WER.
• Performance heavily depends on TTS system choice and speech intelligibility; synthesized speech may introduce artifacts affecting alignment.
• ASR-based baselines drop significantly on document-heavy benchmarks like DocVQA due to transcription errors.

Evidence (verbatim from paper)

For SeedBench, we report the accuracy between the predicted answer and the ground-truth answer. For DocVQA, we report the average normalized Levenshtein similarity (ANLS) between the predicted and ground-truth answers. For MME, we follow LLaVA and report the sum of accuracy scores of the perception tasks. ... we also examine their abilities in speech transcription. We feed the model with only speech and the appropriate textual prompt requesting a transcript and calculate the word-error-rate (WER) for the output.

Citation

@misc{shabtay2025spokenqa,
  title={Spoken question answering for visual queries},
  author={Shabtay et al. (2025)},
  year={2025},
  note={arXiv:2505.23308}
}

• arXiv: 2505.23308