# Refereebench Eval

> Evaluates Multimodal Large Language Models (MLLMs) on automatic sports refereeing tasks, probing their ability to detect incidents, classify fouls, apply sport-specific rules, and ground decisions temporally across 11 different sports. Use when the user wants to benchmark on RefereeBench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/refereebench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/refereebench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/refereebench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/refereebench-eval

---


# refereebench-eval

> RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees — Xu et al. (2026) (arXiv:2604.15736, 2026)

## What this evaluates

Evaluates Multimodal Large Language Models (MLLMs) on automatic sports refereeing tasks, probing their ability to detect incidents, classify fouls, apply sport-specific rules, and ground decisions temporally across 11 different sports.

## Datasets

- **RefereeBench** — total 6475; splits: test (6475)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Computed by directly comparing the model’s predicted answer with the ground truth answer. A match yields 1, otherwise 0. Averaged over all QA pairs.

## Input / output format

**Input**: A full video clip (fixed at 720p resolution, model's default FPS) paired with a multiple-choice question describing a sports officiating scenario.

**Output**: The model's selected multiple-choice answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models tend to over-call fouls on negative samples (legal plays), showing high misidentification rates especially under suggestive wording.
- Performance varies significantly across sports, so a high overall score does not guarantee stable cross-sport generalization.
- Audio modality substantially impacts performance; evaluating with frames-only underestimates model capabilities compared to full video input.

## Evidence (verbatim from paper)

> The accuracy is computed by directly comparing the model’s output with the ground truth answer without the need for any external models.

## Citation

```bibtex
@misc{xu2026refereebench,
  title={RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.15736}
}
```

- arXiv: 2604.15736