Contextiq Retrieval Eval

contextiq-retrieval-eval

ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising — Chaubey et al. (2024) (arXiv:2410.22233, 2024)

What this evaluates

Evaluates zero-shot text-to-video retrieval for contextual advertising. It probes a model's ability to rank relevant video content based on natural language queries using multimodal signals (vision, audio, captions, metadata).

Datasets

• Val-1 — total ?; splits: val (-1)
• Val-2 — total 2000; splits: val (2000)

Metrics

• P@K (primary) — range: [0, 1]
  • Precision at K: (Number of relevant videos in the top K retrieved results) / K. Evaluated at K ∈ {5, 10, 15, 20, 25, 30, 35, 40, 45, 50}.
• Δ_avg — range: [0, 1]
  • Average precision delta across K values: (1/|K|) * Σ_{K∈{5..30}} |P_{V,K} - P_{V+X,K}|, measuring the performance gain of adding a modality X to vision-only.

Input / output format

Input: Natural language text query and video content (or video scenes/clips).

Output: Ranked list of retrieved videos/scenes.

Scoring recipe

# Automated P@K on Val-1
relevant = [v for v in top_k if v in ground_truth]
p_at_k = len(relevant) / k

# Manual validation for Val-2 / Condensed Movies / MSR-VTT
# 3 annotators independently label top-K results as relevant/irrelevant.
final_label = majority_vote(annotator_labels)
p_at_k = sum(final_label) / k

Common pitfalls

• Manual validation relies on a 3-annotator voting system, which may introduce subjectivity or inconsistency for ambiguous queries.
• Internal dataset (Val-2) lacks ground truth, requiring manual annotation for every query, which limits scalability and reproducibility.
• Precision is only reported up to K=50, ignoring long-tail retrieval performance.

Evidence (verbatim from paper)

We use a voting-based system among the annotators to compile the results of our validation. ... Tab. [3] shows the performance of the proposed approach as compared to different approaches on our curated dataset (Val-1 as described in Sec. [4.3]). ... For each query, we search and retrieve the top 30 videos from the dataset. ... The top 30 videos from both methods are then annotated for correctness by 3 annotators. Table 3: Performance comparison on the curated dataset (Val-1) for ContextIQ... P@5 | P@10 | P@15 | P@20 | P@25 | P@30 | P@35 | P@40 | P@45 | P@50

Citation

@misc{chaubey2024contextiq,
  title={ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising},
  author={Chaubey et al. (2024)},
  year={2024},
  note={arXiv:2410.22233}
}

• arXiv: 2410.22233