# Multivent2.0 Eval

> Evaluates event-centric video retrieval across six languages, requiring models to match natural language queries about specific world events to relevant long-form videos using multimodal signals (vision, audio, OCR, metadata). It probes a model's ability to integrate cross-lingual, cross-modal information for complex event understanding rather than simple visual matching. Use when the user wants to benchmark on MultiVENT 2.0, or asks about evaluating this task. Reports Retrieval Performance.

- Skill: `qhjqhj00/multivent2-0-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/multivent2-0-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/multivent2-0-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/multivent2-0-eval

---


# multivent2.0-eval

> MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval — Kriz et al. (2024) (arXiv:2410.11619, 2024)

## What this evaluates

Evaluates event-centric video retrieval across six languages, requiring models to match natural language queries about specific world events to relevant long-form videos using multimodal signals (vision, audio, OCR, metadata). It probes a model's ability to integrate cross-lingual, cross-modal information for complex event understanding rather than simple visual matching.

## Datasets

- **MultiVENT 2.0** — total 218000; splits: test (-1); repo https://github.com/katesanders9/multiVENT

## Metrics

- `Retrieval Performance` **(primary)** — range: [0, 1]
  - Standard video retrieval recall metric. Computes the fraction of queries where the ground-truth video appears in the top-K ranked results. Values range from 0 to 1.

## Input / output format

**Input**: A video (optionally with human-written description) and a natural language query in one of six languages (Arabic, Chinese, English, Korean, Russian, Spanish).

**Output**: A ranked list of candidate videos or similarity scores for each query.

## Scoring recipe

```python
def compute_recall(predictions, gold, k_values=[1, 5, 10]):
    recalls = {}
    for k in k_values:
        hits = sum(1 for pred, gold_id in zip(predictions, gold) if gold_id in pred[:k])
        recalls[f'R@{k}'] = hits / len(predictions)
    return recalls
```

## Common pitfalls

- Videos are long-form, but most VLMs are trained on short segments, causing severe domain mismatch.
- Queries often target non-visual aspects (e.g., audio events, embedded text), making vision-only baselines fail.
- Performance varies significantly by language (Chinese/Korean are hardest) and video type (raw vs. professional).
- Human-written descriptions are not always available for raw/user-generated content, requiring models to handle missing metadata.

## Evidence (verbatim from paper)

> Table [3] presents the performance of these baselines on MultiVENT 2.0, as well as results on MSR-VTT, a standard benchmark video retrieval datasets. A key observation is that while VLMs excel on prior collections, most perform poorly on our task.

## Citation

```bibtex
@misc{kriz2024multivent2,
  title={MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval},
  author={Kriz et al. (2024)},
  year={2024},
  note={arXiv:2410.11619}
}
```

- arXiv: 2410.11619