# Socnern Gar Eval

> socnern-gar-eval

- Skill: `qhjqhj00/socnern-gar-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/socnern-gar-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/socnern-gar-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/socnern-gar-eval

---


# socnern-gar-eval

> Pixels or Positions? Benchmarking Modalities in Group Activity Recognition — Karki et al. (2025) (arXiv:2511.12606, 2025)

## What this evaluates

This benchmark evaluates a model's ability to recognize coordinated group activities in soccer matches by comparing two input modalities: raw video pixels and structured positional tracking data. It probes spatial-temporal reasoning, tactical formation understanding, and robustness to visual shifts by measuring how well models classify 10 distinct group actions from synchronized match footage.

## Datasets

- **SoccerNet-GAR** — total 94285; splits: test (14519), train (-1), val (-1)

## Metrics

- `balanced accuracy` **(primary)** — range: percent
  - Average per-class recall across all 10 activity classes. For each class, recall is computed as TP / (TP + FN). The final score is the arithmetic mean of these 10 recall values, mitigating bias from severe class imbalance.
- `per-class accuracy` — range: percent
  - Standard accuracy computed independently for each of the 10 activity classes (TP / (TP + FP + FN)). Used for detailed ablation and modality comparison.

## Input / output format

**Input**: Two modalities are evaluated: (1) Tracking: frame-level graphs with 23 nodes (8-dimensional entity features) and positional role-based edges (goalkeeper, defender, midfielder, forward). (2) Video: 224×224×3 RGB video frames. Both modalities process synchronized sequences from FIFA World Cup 2022 matches.

**Output**: A single class label from the 10-class vocabulary: PASS, TACKLE, OUT, HEADER, HIGH PASS, THROW IN, CROSS, FREE KICK, SHOT, GOAL.

## Scoring recipe

```python
def compute_balanced_accuracy(predictions, gold_labels, num_classes=10):
    recalls = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        recalls.append(recall)
    return sum(recalls) / num_classes
```

## Common pitfalls

- Severe class imbalance (e.g., GOAL has only 30 test samples) makes standard accuracy misleading; balanced accuracy must be used.
- Confusion between spatially similar actions (e.g., HIGH PASS vs PASS) requires careful per-class analysis rather than relying solely on aggregate scores.
- Tracking models use sentinel coordinates (-2.0) for missing entities, which must be excluded from graph message passing to avoid corrupting representations.

## Evidence (verbatim from paper)

> Metrics. We report balanced accuracy (average per-class recall) as the primary metric due to severe class imbalance.

## Citation

```bibtex
@misc{karki2025pixels,
  title={Pixels or Positions? Benchmarking Modalities in Group Activity Recognition},
  author={Karki et al. (2025)},
  year={2025},
  note={arXiv:2511.12606}
}
```

- arXiv: 2511.12606