Qualcomm Ivd Eval

qualcomm-ivd-eval

Can Vision-Language Models Answer Face to Face Questions in the Real-World? — Pourreza et al. (2025) (arXiv:2503.19356, 2025)

What this evaluates

Probes real-time audio-visual reasoning and situated common sense in dialogue. It requires models to resolve deictic references, perform temporal grounding, and integrate evolving visual and auditory streams to answer open-ended questions posed during video playback.

Datasets

• Qualcomm IVD — total ?; splits: test (-1)

Metrics

• Corr. (primary) — range: [0, 1]
  • Correctness score determined by an LLM judge (Llama3-8B) that evaluates whether a predicted answer matches the ground-truth answer given the question, short answer, and question category.
• BERT — range: [0, 1]
  • Cosine similarity between sentence embeddings of the predicted and ground-truth answers using a pre-trained BERT model.
• METEOR — range: [0, 1]
  • Metric based on exact, stem, synonym, and paraphrase matches between predicted and ground-truth text, weighted by alignment and penalty for fragmentation.
• BLEU — range: [0, 1]
  • Geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty to discourage overly short predictions.
• ROUGE-L — range: [0, 1]
  • F-measure based on the longest common subsequence between the predicted and ground-truth answers, capturing sentence-level fluency and recall.

Input / output format

Input: A video clip trimmed to a specific timestamp (when-to-answer), accompanied by a text question. Optionally includes the raw audio stream for audio-visual models.

Output: Free-form natural language text answering the posed question.

Scoring recipe

def score_answer(question, gt_answer, pred_answer, category):
    # LLM Judge Correctness
    prompt = f'Q: {question}\nGT: {gt_answer}\nPred: {pred_answer}\nCat: {category}\nIs Pred correct?'
    judge_output = llm_judge.generate(prompt)
    corr = 1.0 if 'correct' in judge_output.lower() else 0.0
    
    # Text Similarity Metrics
    bert = bert_similarity(gt_answer, pred_answer)
    meteor = meteor_score_fn(gt_answer, pred_answer)
    bleu = bleu_score_fn(gt_answer, pred_answer)
    rouge_l = rouge_l_score_fn(gt_answer, pred_answer)
    return corr, bert, meteor, bleu, rouge_l

Common pitfalls

• Streaming setup uses ASR-transcribed questions, introducing transcription errors that accumulate and unfairly penalize the model's answering capability.
• Offline setup uses ground-truth questions and timestamps, providing an optimistic estimate of real-world performance that does not reflect streaming ASR inaccuracies.
• The LLM judge's correctness score is prompt-dependent and may not perfectly correlate with human judgment, especially for nuanced or subjective answers.

Evidence (verbatim from paper)

Since the answers in Qualcomm IVD are in free-form, we determine the correctness of an answer using an LLM judge that receives a question, the ground-truth answer, and the predicted answer, alongside the short answer and the category of the question, and determines if the predicted answer is correct. We used a pre-trained Llama3-8B model as the LLM judge. The prompts that were used are provided in the supplementary material. In addition, we report Bert, METEOR, BLEU, and ROUGE scores between the ground-truth answers and the predicted answers.

Citation

@misc{pourreza2025can,
  title={Can Vision-Language Models Answer Face to Face Questions in the Real-World?},
  author={Pourreza et al. (2025)},
  year={2025},
  note={arXiv:2503.19356}
}

• arXiv: 2503.19356