Ilias Eval

ilias-eval

ILIAS: Instance-Level Image retrieval At Scale — Kordopatis-Zilos et al. (2025) (arXiv:2502.11748, 2025)

What this evaluates

Evaluates instance-level image retrieval capability, measuring a model's ability to correctly rank specific object instances within a massive, domain-diverse image corpus. It probes robustness to background clutter, scale variations, and the effectiveness of global versus local descriptors for re-ranking.

Datasets

• ILIAS — total 100000000; splits: test (100000000)

Metrics

• mAP@1k (primary) — range: [0, 100] percent
  • Mean Average Precision at top-1000. Calculates the average of precision values at the rank positions of all relevant retrieved items, capped at k=1000.

Input / output format

Input: A query (image or text) representing a target object instance, and a candidate database of up to 100M images.

Output: A ranked list of the top-1000 candidate images for the query.

Scoring recipe

def compute_map_at_k(retrieved_indices, ground_truth_indices, k=1000):
    retrieved = retrieved_indices[:k]
    relevant = set(ground_truth_indices)
    hits = 0
    ap_sum = 0.0
    for i, idx in enumerate(retrieved):
        if idx in relevant:
            hits += 1
            ap_sum += hits / (i + 1)
    return (ap_sum / len(relevant)) * 100 if relevant else 0.0

Common pitfalls

• Using in-domain fine-tuned models without acknowledging data leakage from the training sets of comparison datasets (e.g., SOP, GLDv2).
• Confusing the 5M mini-ILIAS subset with the full 100M dataset, as performance drops significantly on the full scale.
• Assuming text-to-image and image-to-image retrieval results are highly correlated; the paper shows they are weakly correlated and benefit from ensembling.

Evidence (verbatim from paper)

Table 2. Performance comparison using mAP@1k on ILIAS and mini-ILIAS for global representation models for i2i and t2i.

Citation

@misc{kordopatiszilos2025ilias,
  title={ILIAS: Instance-Level Image retrieval At Scale},
  author={Kordopatis-Zilos et al. (2025)},
  year={2025},
  note={arXiv:2502.11748}
}

• arXiv: 2502.11748