Inquire Eval

inquire-eval

INQUIRE: A Natural World Text-to-Image Retrieval Benchmark — Vendrow et al. (2024) (arXiv:2411.02537, 2024)

What this evaluates

Evaluates text-to-image retrieval capabilities of vision-language models on expert-level, ecologically grounded queries. It probes fine-grained visual understanding, domain-specific language comprehension, and ranking quality across multiple relevant images per query.

Datasets

• INQUIRE — total 250; splits: test (-1), val (-1)

Metrics

• AP@k (mAP@50) (primary) — range: percent
  • Average Precision at k measures precision at each relevant position up to k, averaged over relevant items. The paper uses a modified normalization factor suited to retrieval settings with multiple relevant images per query.
• nDCG@k — range: percent
  • Normalized Discounted Cumulative Gain discounts relevance scores by their position rank, normalized by the ideal DCG.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank is the average of the reciprocal of the rank of the first relevant item in the retrieved list.

Input / output format

Input: Text query; for the rerank task, a fixed candidate set of top-100 images is provided alongside the query.

Output: Ranked list of images (or relevance scores/rankings for reranking).

Scoring recipe

def compute_ap_at_k(retrieved, relevant, k):
    hits = 0
    prec_sum = 0.0
    for i, img in enumerate(retrieved[:k]):
        if img in relevant:
            hits += 1
            prec_sum += hits / (i + 1)
    # Paper uses a modified normalization factor for multi-relevant retrieval
    norm = min(k, len(relevant))
    return (prec_sum / norm) * 100

Common pitfalls

• Using recall@k instead of AP/nDCG/MRR, which ignores ranking quality and fails when multiple images are relevant.
• Assuming standard AP normalization applies; the paper explicitly notes a modified normalization factor for this multi-relevant-image setting.
• Expecting high scores on the fullrank task; even optimal reranking only reaches ~50-65 mAP@50, highlighting the benchmark's difficulty.

Evidence (verbatim from paper)

We evaluate using Average Precision at k (AP@k), Normalized Discounted Cumulative Gain (nDCG), and Mean Reciprocal Rank (MRR). We primarily discuss AP as we find that this metric is the most discriminative of model performance. While these metrics have been commonly used to evaluate text retrieval, especially in the context of large-scale document retrieval, they have not found use in image retrieval due to the nonexistence of benchmarks like Inquire containing many relevant images for retrieval, rather than just one. Thus, we include them in our analysis to encourage their use in future image retrieval research. We note that the utilized AP@k metric uses a modified normalization factor suited to the retrieval setting.

Citation

@misc{vendrow2024inquire,
  title={INQUIRE: A Natural World Text-to-Image Retrieval Benchmark},
  author={Vendrow et al. (2024)},
  year={2024},
  note={arXiv:2411.02537}
}

• arXiv: 2411.02537