Sgmri Vqa Eval

sgmri-vqa-eval

Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI — Moukheiber et al. (2026) (arXiv:2604.15808, 2026)

What this evaluates

Evaluates vision-language models on multi-frame spatial reasoning and grounding in volumetric MRI scans. It probes the model's ability to answer clinical questions, generate free-text reasoning, and accurately localize anatomical findings across single slices or full 3D volumes.

Datasets

• SGMRI-VQA — total 41307; splits: train (-1), val (-1); repo https://github.com/lamawmouk/SGMRI-VQA

Metrics

• A-Score (primary) — range: [0, 1]
  • Measures factual answer accuracy. Uses exact match for Yes/No and single-choice questions, F1 over the selected option set for multiple-choice, and the average of keyword recall and semantic similarity (via SentenceTransformer embeddings) for open-ended questions.
• AR-Score — range: [0, 1]
  • Evaluates free-text clinical reasoning quality as a weighted combination of GPT-4o-mini judge scoring (0.4), BERTScore F1 (0.2), smoothed BLEU (0.2), and ROUGE-L F1 (0.2). For localization tasks, bounding box coordinates and frame references are stripped from both prediction and reference before scoring.
• V-Score — range: [0, 1]
  • Measures pixel-level spatial grounding accuracy as the mean Intersection over Union (IoU) between predicted and ground-truth bounding boxes. Predicted boxes are extracted via a parser, matched to ground truth using frame-aware IoU matrices and the Hungarian algorithm.

Input / output format

Input: Multi-frame sequence of MRI slices (volume-level) or a single MRI slice (image-level) paired with a clinical question.

Output: Textual answer (and reasoning) and/or bounding box coordinates with frame indices for localization tasks.

Scoring recipe

def compute_ascore(pred, gold, q_type):
    if q_type in ['yes_no', 'single_choice']: return 1.0 if pred == gold else 0.0
    elif q_type == 'multiple_choice':
        p, g = set(pred), set(gold)
        return (2*len(p&g))/(len(p)+len(g)) if (len(p)+len(g))>0 else 0.0
    else: return 0.5*keyword_recall(pred, gold) + 0.5*st_similarity(pred, gold)

def compute_ar_score(pred, gold, task):
    if task == 'localization':
        pred, gold = strip_coords_frames(pred), strip_coords_frames(gold)
    return 0.4*gpt_judge(pred, gold) + 0.2*bertscore_f1(pred, gold) + 0.2*bleu(pred, gold) + 0.2*rouge(pred, gold)

def compute_v_score(pred_boxes, gold_boxes):
    ious = hungarian_match(pred_boxes, gold_boxes, metric='frame_aware_iou')
    return mean(ious)

Common pitfalls

• For localization tasks, AR-Score explicitly strips bounding box coordinates and frame references before scoring to evaluate anatomical description quality rather than numeric coordinates.
• V-Score requires frame-aware IoU matching via the Hungarian algorithm; naive per-frame IoU or ignoring frame indices will yield incorrect scores.
• A-Score scoring rules change based on question format (exact match vs. F1 vs. embedding similarity); applying a single rule across all types will misrepresent performance.

Evidence (verbatim from paper)

We use three complementary evaluation metrics. A-Score measures factual answer accuracy for detection, counting, classification, and diagnosis tasks. Scoring differs by question format: closed-ended questions use exact match on Yes/No, single-choice uses exact match on the selected option letter, multiple-choice uses F1 over the selected option set, and open-ended uses the average of keyword recall and semantic similarity via SentenceTransformer embeddings. AR-Score evaluates free-text clinical reasoning quality for captioning and localization tasks as a weighted combination of GPT-4o-mini judge scoring (weight 0.4), BERTScore F1 (0.2), smoothed BLEU (0.2), and ROUGE-L F1 (0.2). For localization tasks, bounding box coordinates and frame references are stripped from both prediction and reference before scoring, so the judge evaluates anatomical description quality rather than numeric coordinates. V-Score measures pixel-level spatial grounding accuracy as mean IoU between predicted and ground-truth bounding boxes for localization tasks, complementing the textual anatomical descriptions evaluated by AR-Score—together, these two metrics jointly assess where anatomically (in text) and w

Citation

@misc{moukheiber2026beyond,
  title={Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI},
  author={Moukheiber et al. (2026)},
  year={2026},
  note={arXiv:2604.15808}
}

• arXiv: 2604.15808