Sports QA Eval

sports-qa-eval

Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports — Li et al. (2024) (arXiv:2401.01505, 2024)

What this evaluates

Probes video question answering capabilities, specifically focusing on temporal reasoning, action causality, counterfactual inference, and fine-grained motion understanding within professional sports contexts.

Datasets

• Sports-QA — total 94000; splits: train (-1), val (-1), test (-1); repo https://github.com/HopLee6/Sports-QA

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answer classes out of the total number of questions.
• F1-score — range: percent
  • Harmonic mean of precision and recall, computed to account for the long-tail distribution and class imbalance across the 191 answer categories.

Input / output format

Input: Down-sampled video frames (5 FPS) with pre-extracted global/local appearance and motion features, paired with a natural language question.

Output: A single answer string selected from a fixed vocabulary of 191 classes.

Scoring recipe

correct = sum(1 for p, g in zip(predictions, gold) if p == g)
accuracy = (correct / len(predictions)) * 100
# Macro-averaged F1 to handle class imbalance
f1 = f1_score(gold, predictions, average='macro') * 100

Common pitfalls

• The dataset contains 191 answer classes with a long-tail distribution, so accuracy alone can be misleading; F1-score is required to properly evaluate performance on rare classes.
• Baseline models like BlindQA achieve high accuracy by exploiting question-answer semantic correlations without using visual features, making it crucial to verify visual grounding.
• Visual features are pre-extracted using Faster-RCNN and I3D rather than raw video inputs, which may limit direct comparison with end-to-end video-language models.

Evidence (verbatim from paper)

For evaluation metrics, we employ accuracy to showcase the models' performance on our dataset. Additionally, recognizing the long-tail distribution of our dataset, we evaluate methods using F1-score. F1-score is particularly important for sports VideoQA as it explicitly reflects performance averaged over imbalanced classes.

Citation

@misc{li2024sportsqa,
  title={Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2401.01505}
}

• arXiv: 2401.01505