Socnern Gar Eval

socnern-gar-eval

Pixels or Positions? Benchmarking Modalities in Group Activity Recognition — Karki et al. (2025) (arXiv:2511.12606, 2025)

What this evaluates

This benchmark evaluates a model's ability to recognize coordinated group activities in soccer matches by comparing two input modalities: raw video pixels and structured positional tracking data. It probes spatial-temporal reasoning, tactical formation understanding, and robustness to visual shifts by measuring how well models classify 10 distinct group actions from synchronized match footage.

Datasets

• SoccerNet-GAR — total 94285; splits: test (14519), train (-1), val (-1)

Metrics

• balanced accuracy (primary) — range: percent
  • Average per-class recall across all 10 activity classes. For each class, recall is computed as TP / (TP + FN). The final score is the arithmetic mean of these 10 recall values, mitigating bias from severe class imbalance.
• per-class accuracy — range: percent
  • Standard accuracy computed independently for each of the 10 activity classes (TP / (TP + FP + FN)). Used for detailed ablation and modality comparison.

Input / output format

Input: Two modalities are evaluated: (1) Tracking: frame-level graphs with 23 nodes (8-dimensional entity features) and positional role-based edges (goalkeeper, defender, midfielder, forward). (2) Video: 224×224×3 RGB video frames. Both modalities process synchronized sequences from FIFA World Cup 2022 matches.

Output: A single class label from the 10-class vocabulary: PASS, TACKLE, OUT, HEADER, HIGH PASS, THROW IN, CROSS, FREE KICK, SHOT, GOAL.

Scoring recipe

def compute_balanced_accuracy(predictions, gold_labels, num_classes=10):
    recalls = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        recalls.append(recall)
    return sum(recalls) / num_classes

Common pitfalls

• Severe class imbalance (e.g., GOAL has only 30 test samples) makes standard accuracy misleading; balanced accuracy must be used.
• Confusion between spatially similar actions (e.g., HIGH PASS vs PASS) requires careful per-class analysis rather than relying solely on aggregate scores.
• Tracking models use sentinel coordinates (-2.0) for missing entities, which must be excluded from graph message passing to avoid corrupting representations.

Evidence (verbatim from paper)

Metrics. We report balanced accuracy (average per-class recall) as the primary metric due to severe class imbalance.

Citation

@misc{karki2025pixels,
  title={Pixels or Positions? Benchmarking Modalities in Group Activity Recognition},
  author={Karki et al. (2025)},
  year={2025},
  note={arXiv:2511.12606}
}

• arXiv: 2511.12606