Severe Benchmark Eval

severe-benchmark-eval

How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning? — Thoker et al. (2022) (arXiv:2203.14221, 2022)

What this evaluates

Evaluates the generalization and sensitivity of video self-supervised learning models to domain shifts, downstream sample sizes, action similarity, and task shifts beyond action recognition.

Datasets

• UCF-101 — total ?; splits: test (-1)
• NTU-60 — total ?; splits: test (-1)
• FineGym (Gym-99) — total ?; splits: test (-1)
• Something-Something-v2 — total ?; splits: test (-1)
• EPIC-Kitchens-100 — total ?; splits: test (-1)
• Charades — total ?; splits: test (-1)
• AVA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Top-1 classification accuracy computed over the test split after finetuning a task-dependent head on the pre-trained R(2+1)D-18 backbone.

Input / output format

Input: Video clips/frames from downstream datasets, processed through a pre-trained R(2+1)D-18 backbone.

Output: Class labels (or action categories) predicted by a task-dependent head attached to the backbone.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Assuming self-supervised pre-training always outperforms supervised pre-training across all shifts.
• Overlooking the impact of domain/task shifts by only evaluating on datasets similar to Kinetics-400.
• Not controlling for hyperparameters during finetuning across different pre-trained models.

Evidence (verbatim from paper)

To finetune for downstream tasks we simply attach a task-dependent head at the last layer of the pre-trained R(2+1)D-18 backbone to produce label predictions for the corresponding task. For a fair comparison, we use the same set of hyper-parameters, optimization and pre-processing during the downstream training of each model.

Citation

@misc{thoker2022severe,
  title={How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?},
  author={Thoker et al. (2022)},
  year={2022},
  note={arXiv:2203.14221}
}

• arXiv: 2203.14221