# Omniret Retrieval Eval

> Evaluates multimodal retrieval capabilities across text, image, video, and audio modalities, including composed queries. It probes the model's ability to align heterogeneous media types and rank relevant candidates under varying modality combinations. Use when the user wants to benchmark on Extended M-BEIR, MMEBv2, ACM (Audio-Centric Multimodal Benchmark), or asks about evaluating this task. Reports Recall@5.

- Skill: `qhjqhj00/omniret-retrieval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/omniret-retrieval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/omniret-retrieval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/omniret-retrieval-eval

---


# omniret-retrieval-eval

> OmniRet: Efficient and High-Fidelity Omni Modality Retrieval — Chuong Huynh et al. (2026) (arXiv:2603.02098, 2026)

## What this evaluates

Evaluates multimodal retrieval capabilities across text, image, video, and audio modalities, including composed queries. It probes the model's ability to align heterogeneous media types and rank relevant candidates under varying modality combinations.

## Datasets

- **Extended M-BEIR** — total ?; splits: test (-1)
- **MMEBv2** — total ?; splits: test (-1)
- **ACM (Audio-Centric Multimodal Benchmark)** — total ?; splits: test (-1)

## Metrics

- `Recall@5` **(primary)** — range: percent
  - Recall@k measures the fraction of queries where the ground-truth relevant item appears in the top-k retrieved results. The paper primarily reports Recall@5, with Recall@1 for MMEBv2 and Recall@10 for FashionIQ/Fashion200K.

## Input / output format

**Input**: Multimodal query (text, image, video, audio, or combinations like audio+text) paired with a candidate pool of media items.

**Output**: Ranked list of candidate media items based on embedding similarity scores.

## Scoring recipe

```python
def compute_recall_at_k(retrieved_indices, relevant_indices, k=5):
    relevant_set = set(relevant_indices)
    top_k = retrieved_indices[:k]
    hits = len(relevant_set.intersection(top_k))
    return hits / len(relevant_set) if relevant_set else 0.0

# Average across all queries in the benchmark
```

## Common pitfalls

- Do not use a global candidate pool across all benchmarks; the protocol requires testing each benchmark individually and averaging results per group.
- k values vary by dataset: use k=5 for most, k=10 for FashionIQ/Fashion200K, and k=1 for MMEBv2.
- Ensure modality-specific encoders (SigLIP for vision, QwenAudio for audio) are correctly aligned before retrieval to avoid modality mismatch.

## Evidence (verbatim from paper)

> Following MMEBv2, we test each benchmark individually and average the results for each group instead of using a global candidate pool. We use Recall@$k$ with $k\=5$ for most benchmarks, except for FashionIQ*[[73](#bib.bib36 "Fashion iq: a new dataset towards retrieving images by natural language feedback")]* and Fashion200K*[[18](#bib.bib14 "Automatic spatially-aware fashion concept discovery")]*, which use $k\=10$. On MMEBv2, we compare performance on classification (CLS), retrieval (RET), and video moment retrieval tasks (MRET). Recall@1 is used as the metric for MMEBv2, while on our proposed benchmark, we report Recall@5.

## Citation

```bibtex
@misc{huynh2026omniret,
  title={OmniRet: Efficient and High-Fidelity Omni Modality Retrieval},
  author={Chuong Huynh et al. (2026)},
  year={2026},
  note={arXiv:2603.02098}
}
```

- arXiv: 2603.02098