Human Action Recognition Eval

human-action-recognition-eval

Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos — Yang Qian et al. (2024) (arXiv:2402.08875, 2024)

What this evaluates

Evaluates a video foundation model's ability to recognize and classify human actions across diverse, real-world, and benchmark video datasets. It tests generalization from self-supervised pre-training on unstructured social media content to structured action recognition tasks.

Datasets

• Kinetics-400 — total ?; splits: test (-1)
• Something-Something V2 — total ?; splits: test (-1)
• UCF-101 — total ?; splits: test (-1)
• HMDB51 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Fraction of correctly predicted action classes out of the total number of samples in the evaluation set, multiplied by 100.

Input / output format

Input: Video clips sampled as a sequence of 16 frames from raw videos, resized to 224x224 pixels. Sampling stride varies by dataset (τ=4 for Kinetics-400, τ=2 for Something-Something V2).

Output: Predicted action class label from the dataset's predefined category set.

Scoring recipe

correct = 0
total = 0
for video, true_label in dataset:
    frames = sample_frames(video, num_frames=16, stride=dataset_stride)
    frames = resize(frames, 224, 224)
    pred_label = model(frames)
    if pred_label == true_label:
        correct += 1
    total += 1
return (correct / total) * 100

Common pitfalls

• Confusing the pre-training dataset (TikTok, 283k unlabeled videos) with the evaluation benchmarks.
• Different fine-tuning protocols are used: direct fine-tuning for Kinetics-400 and Something-Something V2, versus transfer learning (K400 -> UCF101/HMDB51) for the latter two.
• Model capacity (ViT-Base vs ViT-Giant) drastically changes results; reporting without specifying backbone is misleading.
• Frame sampling stride differs across datasets (τ=4 vs τ=2), affecting temporal resolution and performance.

Evidence (verbatim from paper)

We report the fine-tuning accuracy of multiple view fusion $(5 imes 3)$ . Both models are pre-trained and fine-tuned at the input of $16 imes 224 imes 224$ and sampling stride $ au = 4$ .

Citation

@misc{qian2024advancing,
  title={Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos},
  author={Yang Qian et al. (2024)},
  year={2024},
  note={arXiv:2402.08875}
}

• arXiv: 2402.08875