Vl Jepa Zero Shot Benchmarks

vl-jepa-zero-shot-benchmarks

Action100M: A Large-scale Video Action Dataset — Chen et al. (2026) (arXiv:2601.10592, 2026)

What this evaluates

Evaluates zero-shot video understanding capabilities on action recognition and text-to-video retrieval tasks across diverse benchmarks.

Datasets

• Something-something-v2 (SSv2) — total ?; splits: test (-1)
• EPIC-KITCHENS-100 (EK-100) — total ?; splits: test (-1)
• EgoExo4D Keysteps — total ?; splits: test (-1)
• Kinetics-400 — total ?; splits: test (-1)
• COIN — total ?; splits: test (-1)
• CrossTask — total ?; splits: test (-1)
• MSR-VTT — total ?; splits: test (-1)
• ActivityNet — total ?; splits: test (-1)
• DiDeMo — total ?; splits: test (-1)
• MSVD — total ?; splits: test (-1)
• YouCook2 — total ?; splits: test (-1)
• PVD-Bench — total ?; splits: test (-1)
• Dream-1K — total ?; splits: test (-1)
• VDC-1K — total ?; splits: test (-1)

Metrics

• top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels out of the total number of instances.
• recall@1 (primary) — range: [0, 1]
  • Fraction of queries where the ground-truth video is ranked first in the retrieved list.

Input / output format

Input: Video frames sampled at 1, 8, or 32 frames per input, paired with class labels or text queries.

Output: Predicted class label for action recognition; ranked list of videos for retrieval.

Scoring recipe

def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    r1 = sum(1 for p, g in zip(predictions, golds) if p[0] == g) / len(golds)
    return {'top-1_accuracy': acc, 'recall@1': r1}

Common pitfalls

• COIN and CrossTask require separate evaluation for segment-level step recognition and global task recognition.
• Zero-shot evaluation implies no fine-tuning on target benchmarks, only inference.
• Frame sampling strategy (1, 8, or 32 frames) significantly impacts memory and performance, requiring consistent settings across models.

Evidence (verbatim from paper)

We evaluate on two tasks: Zero-shot action recognition (top-1 accuracy) on eight benchmarks: Something-something-v2 (SSv2) (Goyal et al., [2017]), EPIC-KITCHENS-100 (EK-100) (Damen et al., [2020]), EgoExo4D Keysteps (Grauman et al., [2024]), Kinetics-400 (Kay et al., [2017]), COIN (Tang et al., [2019]), and CrossTask (Zhukov et al., [2019]). For COIN and Crosstask, we evaluate both segment-level step recognition and global task recognition. Zero-shot text-to-video retrieval (recall@1) on eight benchmarks: MSR-VTT (Xu et al., [2016]), ActivityNet (Caba Heilbron et al., [2015]), DiDeMo (Anne Hendricks et al., [2017]), MSVD (Chen and Dolan, [2011]), YouCook2 (Zhou et al., [2018]), PVD-Bench (Bolya et al., [2025]), Dream-1K (Wang et al., [2024a]), and VDC-1K (Chai et al., [2024]).

Citation

@misc{chen2026action100m,
  title={Action100M: A Large-scale Video Action Dataset},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2601.10592}
}

• arXiv: 2601.10592