Rextthewild Eval

rextthewild-eval

ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding — Banerjee et al. (2026) (arXiv:2603.19517, 2026)

What this evaluates

Evaluates multimodal models' ability to understand real-world medical photographs by answering clinician-verified multiple-choice questions across seven clinical domains. It probes capabilities in geometric perception, anatomical localization, clinical characterization, and causal reasoning.

Datasets

• ReXInTheWild — total 955; splits: test (955)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions out of the total number of questions. Reported with 95% confidence intervals.

Input / output format

Input: A real-world medical photograph paired with a multiple-choice question.

Output: Selection of the correct option from the provided multiple-choice answers.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Models frequently confuse left/right sides of the body or flexed/extended joints (geometric errors).
• Causality errors often co-occur with lower-level geometric or localization mistakes, making it hard to isolate reasoning failures.
• Performance varies significantly across clinical domains (e.g., 'Eyes' vs 'Head & Neck'), so aggregate accuracy may mask domain-specific weaknesses.

Evidence (verbatim from paper)

Gemini-3 achieved the strongest performance with 78% accuracy (95% CI: 0.75, 0.81) across all questions, followed by Claude Opus 4.5 with 72% accuracy (95% CI: 0.69, 0.75) and GPT-5 with 68% accuracy (95% CI: 0.65, 0.71).

Citation

@misc{banerjee2026rextthewild,
  title={ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding},
  author={Banerjee et al. (2026)},
  year={2026},
  note={arXiv:2603.19517}
}

• arXiv: 2603.19517