Omniret Retrieval Eval

omniret-retrieval-eval

OmniRet: Efficient and High-Fidelity Omni Modality Retrieval — Chuong Huynh et al. (2026) (arXiv:2603.02098, 2026)

What this evaluates

Evaluates multimodal retrieval capabilities across text, image, video, and audio modalities, including composed queries. It probes the model's ability to align heterogeneous media types and rank relevant candidates under varying modality combinations.

Datasets

• Extended M-BEIR — total ?; splits: test (-1)
• MMEBv2 — total ?; splits: test (-1)
• ACM (Audio-Centric Multimodal Benchmark) — total ?; splits: test (-1)

Metrics

• Recall@5 (primary) — range: percent
  • Recall@k measures the fraction of queries where the ground-truth relevant item appears in the top-k retrieved results. The paper primarily reports Recall@5, with Recall@1 for MMEBv2 and Recall@10 for FashionIQ/Fashion200K.

Input / output format

Input: Multimodal query (text, image, video, audio, or combinations like audio+text) paired with a candidate pool of media items.

Output: Ranked list of candidate media items based on embedding similarity scores.

Scoring recipe

def compute_recall_at_k(retrieved_indices, relevant_indices, k=5):
    relevant_set = set(relevant_indices)
    top_k = retrieved_indices[:k]
    hits = len(relevant_set.intersection(top_k))
    return hits / len(relevant_set) if relevant_set else 0.0

# Average across all queries in the benchmark

Common pitfalls

• Do not use a global candidate pool across all benchmarks; the protocol requires testing each benchmark individually and averaging results per group.
• k values vary by dataset: use k=5 for most, k=10 for FashionIQ/Fashion200K, and k=1 for MMEBv2.
• Ensure modality-specific encoders (SigLIP for vision, QwenAudio for audio) are correctly aligned before retrieval to avoid modality mismatch.

Evidence (verbatim from paper)

Following MMEBv2, we test each benchmark individually and average the results for each group instead of using a global candidate pool. We use Recall@$k$ with $k=5$ for most benchmarks, except for FashionIQ*[73]* and Fashion200K*[18]*, which use $k=10$. On MMEBv2, we compare performance on classification (CLS), retrieval (RET), and video moment retrieval tasks (MRET). Recall@1 is used as the metric for MMEBv2, while on our proposed benchmark, we report Recall@5.

Citation

@misc{huynh2026omniret,
  title={OmniRet: Efficient and High-Fidelity Omni Modality Retrieval},
  author={Chuong Huynh et al. (2026)},
  year={2026},
  note={arXiv:2603.02098}
}

• arXiv: 2603.02098