Soccerchat Eval

soccerchat-eval

SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding — Gautam et al. (2025) (arXiv:2505.16630, 2025)

What this evaluates

Evaluates multimodal video-language models on soccer-specific tasks: referee decision validation via question-answering and multi-label action classification. It probes the model's ability to align visual, auditory, and textual cues with ground-truth soccer events and rules.

Datasets

• XFoul validation dataset — total ?; splits: test (-1)
• SoccerNet-v2 — total ?; splits: test (-1)

Metrics

• QwQ Scorer (primary) — range: [0, 10]
  • A custom LLM-based evaluator that assigns a score between 0 and 10 based on the alignment between the model's generated answer and the ground truth annotation.
• F1 Score (wt) (primary) — range: [0, 1]
  • Weighted F1-score for multi-label classification, averaging the F1 across classes weighted by their support (number of true instances).
• Cohen Kappa — range: [0, 1]
  • Statistical measure of inter-rater agreement for qualitative (categorical) items, adjusted for chance.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient, a balanced measure of binary/multiclass classification quality that takes true/false positives and negatives into account.
• Hamming Loss — range: [0, 1]
  • Fraction of labels that are incorrectly predicted, averaged over all samples and labels.

Input / output format

Input: Multimodal soccer match clips (video, audio, text) with jersey color annotations and ASR transcripts, paired with questions or event labels.

Output: Textual answers for QA tasks; class labels (6-class or 16-class) for action classification tasks.

Scoring recipe

# QwQ Scorer (QA/Referee Task)
# Returns float 0-10 based on LLM alignment with ground truth
score = qwq_model.evaluate(prediction=answer, gold=ground_truth)

# Action Classification Metrics (6-class or 16-class)
precision = precision_score(y_true, y_pred, average='weighted')
recall = recall_score(y_true, y_pred, average='weighted')
f1 = f1_score(y_true, y_pred, average='weighted')
kappa = cohen_kappa_score(y_true, y_pred)
mcc = matthews_corrcoef(y_true, y_pred)
hamming = hamming_loss(y_true, y_pred)

Common pitfalls

• The QwQ Scorer is a custom LLM-based evaluator; exact scores depend on the specific model version and prompt template, making exact replication difficult without the inference code.
• Classification metrics use weighted averages, but the paper does not specify how multi-label predictions are thresholded or how class imbalance is handled during evaluation.
• Fine-tuning sequence (pretrain then fine-tune vs. joint training) drastically impacts performance, contrary to typical transfer learning expectations, requiring careful dataset mixing strategies.

Evidence (verbatim from paper)

The evaluation relied on the QwQ Scorer model, which assigns scores between 0 and 10 based on alignment with ground truth annotations. Performance comparisons were visualized using grouped violin plots to highlight score distributions across different model configurations.

Citation

@misc{gautam2025soccerchat,
  title={SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding},
  author={Gautam et al. (2025)},
  year={2025},
  note={arXiv:2505.16630}
}

• arXiv: 2505.16630