Multivent2.0 Eval

multivent2.0-eval

MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval — Kriz et al. (2024) (arXiv:2410.11619, 2024)

What this evaluates

Evaluates event-centric video retrieval across six languages, requiring models to match natural language queries about specific world events to relevant long-form videos using multimodal signals (vision, audio, OCR, metadata). It probes a model's ability to integrate cross-lingual, cross-modal information for complex event understanding rather than simple visual matching.

Datasets

• MultiVENT 2.0 — total 218000; splits: test (-1); repo https://github.com/katesanders9/multiVENT

Metrics

• Retrieval Performance (primary) — range: [0, 1]
  • Standard video retrieval recall metric. Computes the fraction of queries where the ground-truth video appears in the top-K ranked results. Values range from 0 to 1.

Input / output format

Input: A video (optionally with human-written description) and a natural language query in one of six languages (Arabic, Chinese, English, Korean, Russian, Spanish).

Output: A ranked list of candidate videos or similarity scores for each query.

Scoring recipe

def compute_recall(predictions, gold, k_values=[1, 5, 10]):
    recalls = {}
    for k in k_values:
        hits = sum(1 for pred, gold_id in zip(predictions, gold) if gold_id in pred[:k])
        recalls[f'R@{k}'] = hits / len(predictions)
    return recalls

Common pitfalls

• Videos are long-form, but most VLMs are trained on short segments, causing severe domain mismatch.
• Queries often target non-visual aspects (e.g., audio events, embedded text), making vision-only baselines fail.
• Performance varies significantly by language (Chinese/Korean are hardest) and video type (raw vs. professional).
• Human-written descriptions are not always available for raw/user-generated content, requiring models to handle missing metadata.

Evidence (verbatim from paper)

Table [3] presents the performance of these baselines on MultiVENT 2.0, as well as results on MSR-VTT, a standard benchmark video retrieval datasets. A key observation is that while VLMs excel on prior collections, most perform poorly on our task.

Citation

@misc{kriz2024multivent2,
  title={MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval},
  author={Kriz et al. (2024)},
  year={2024},
  note={arXiv:2410.11619}
}

• arXiv: 2410.11619