Rexsenovqa Eval

rexsenovqa-eval

ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding — Wang et al. (2026) (arXiv:2604.10916, 2026)

What this evaluates

This benchmark evaluates video-language models on procedure-centric ultrasound understanding, specifically probing dynamic procedural reasoning, causal troubleshooting, and temporal action understanding. It measures how well models interpret visual evidence and reason through medical imaging procedures without relying on audio or static priors.

Datasets

• ReXSonoVQA — total ?; splits: test (-1); repo https://github.com/rajpurkarlab/RexSonoVQA

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy for multiple-choice questions, calculated as the fraction of predictions matching the ground-truth option.
• mean_score_0_2 — range: other
  • LLM-as-a-judge score on a 0–2 rubric for free-response items. Score 2 requires correct visual evidence and reasoning; Score 1 requires one correct and one incorrect; Score 0 requires both incorrect.

Input / output format

Input: Video clip (audio removed) spanning [time_start, time_end] plus the question and MCQ options (if applicable). A text-only baseline setting provides only the question and options without visual input.

Output: For MCQ: a single option label. For free-response: a natural language explanation containing visual evidence and procedural reasoning/conclusion.

Scoring recipe

def score_mcq(pred, gold):
    return 1.0 if pred == gold else 0.0

def score_free(pred, gold, judge):
    # Judge applies 0-2 rubric: 2=both correct, 1=one correct, 0=both incorrect
    return judge.evaluate(pred, gold, rubric="0-2")

def compute_metrics(preds, golds, is_mcq):
    if is_mcq:
        return sum(score_mcq(p, g) for p, g in zip(preds, golds)) / len(golds)
    return sum(score_free(p, g, judge) for p, g in zip(preds, golds)) / len(golds)

Common pitfalls

• Audio tracks are explicitly removed from all video clips, so models relying on sonographer narration or audio cues will be unfairly penalized.
• The LLM-as-a-judge protocol requires careful prompt engineering to avoid bias toward verbose answers or over-penalizing minor phrasing differences.
• The text-only baseline setting is used diagnostically; questions with high text-only accuracy may suffer from answer-leaking or generic priors rather than true visual reasoning.

Evidence (verbatim from paper)

MCQ items are scored using accuracy by comparing the selected option against the ground-truth option. Free-response items are graded using an LLM-as-a-judge protocol with a three-level rubric: Score 2 (Correct): Both visual evidence and procedural reasoning/conclusion are correct. Score 1 (Partially Correct): Either the visual evidence is correct but the reasoning/conclusion is wrong, or the reasoning/conclusion is correct but the visual evidence is incorrect. Score 0 (Incorrect): Both visual evidence and reasoning/conclusion are incorrect.

Citation

@misc{wang2026rexsenovqa,
  title={ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.10916}
}

• arXiv: 2604.10916