Refereebench Eval

refereebench-eval

RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees — Xu et al. (2026) (arXiv:2604.15736, 2026)

What this evaluates

Evaluates Multimodal Large Language Models (MLLMs) on automatic sports refereeing tasks, probing their ability to detect incidents, classify fouls, apply sport-specific rules, and ground decisions temporally across 11 different sports.

Datasets

• RefereeBench — total 6475; splits: test (6475)

Metrics

• accuracy (primary) — range: percent
  • Computed by directly comparing the model’s predicted answer with the ground truth answer. A match yields 1, otherwise 0. Averaged over all QA pairs.

Input / output format

Input: A full video clip (fixed at 720p resolution, model's default FPS) paired with a multiple-choice question describing a sports officiating scenario.

Output: The model's selected multiple-choice answer.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• Models tend to over-call fouls on negative samples (legal plays), showing high misidentification rates especially under suggestive wording.
• Performance varies significantly across sports, so a high overall score does not guarantee stable cross-sport generalization.
• Audio modality substantially impacts performance; evaluating with frames-only underestimates model capabilities compared to full video input.

Evidence (verbatim from paper)

The accuracy is computed by directly comparing the model’s output with the ground truth answer without the need for any external models.

Citation

@misc{xu2026refereebench,
  title={RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.15736}
}

• arXiv: 2604.15736