Uvrb Eval

uvrb-eval

Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum — Zhuoning Guo et al. (arXiv:2510.27571, 2025)

What this evaluates

Evaluates zero-shot generalization of video embedding models across 16 diverse retrieval tasks and domains. It probes capabilities like spatial/temporal reasoning, compositional understanding, and partially relevant matching, revealing how well models generalize beyond standard benchmarks.

Datasets

• UVRB (Universal Video Retrieval Benchmark) — total ?; splits: test (-1)

Metrics

• Recall@1 (R@1) (primary) — range: [0, 1]
  • Fraction of queries where the ground-truth video ranks first among all candidates based on cosine similarity.
• Recall@10 (R@10) — range: [0, 1]
  • Fraction of queries where the ground-truth video ranks within the top 10 candidates. Used for challenging datasets with fuzzy queries.
• Precision@1 (P@1) — range: [0, 1]
  • Fraction of queries where at least one ground-truth video ranks first among candidates. Used for datasets with multiple positive matches.

Input / output format

Input: Text query paired with a video uniformly sampled into exactly 8 raw visual frames. Audio, speech, and metadata are excluded.

Output: Normalized embedding vector (bf16 precision). Cosine similarity is computed between query and video embeddings for ranking.

Scoring recipe

def compute_recall_at_k(sim_matrix, labels, k=1):
    # sim_matrix: (N, N) cosine similarities (embeddings already L2-normalized)
    # labels: (N,) ground truth video indices for each query
    ranks = np.argsort(-sim_matrix, axis=1)
    top_k_indices = ranks[:, :k]
    hits = np.any(top_k_indices == labels, axis=1)
    return np.mean(hits)

Common pitfalls

• Audio, speech, and metadata are strictly excluded; only raw visual frames are used.
• Each video is uniformly sampled to exactly 8 frames regardless of original length.
• Baseline models may have seen in-domain test data, creating an unfair advantage not present for the proposed GVE model.
• Frame resolution and token limits are adaptively adjusted per model to enforce a <200 visual tokens/frame constraint for MLLM-based models.

Evidence (verbatim from paper)

Our primary metric is Recall@1 (R@1), which measures if the most relevant item is the correct one. For more challenging datasets with fuzzy queries (e.g., CMRB and LoVR-TH), we choose to report Recall@10 (R@10). Additionally, we use Precision@1 (P@1) for the MS-TI and MS-TV with multiple positive candidates.

Citation

@misc{guo2025towardsuniversalvideoretrieval,
  title={Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum},
  author={Zhuoning Guo et al.},
  year={2025},
  note={arXiv:2510.27571}
}

• arXiv: 2510.27571