Image Text Retrieval Eval

image-text-retrieval-eval

HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval — Chen et al. (2022) (arXiv:2205.12105, 2022)

What this evaluates

Evaluates a model's ability to retrieve relevant images given a text query and vice versa. It probes cross-modal alignment and ranking capabilities under both standard test-set and large-scale candidate-pool settings.

Datasets

• Flickr30k — total 31000; splits: train (29000), val (1000), test (1000)
• COCO — total 123000; splits: train (114000), val (5000), test (5000)

Metrics

• Recall@K (R@K) (primary) — range: percent
  • For each query, rank candidates by similarity score. R@K equals 1 if at least one ground-truth match appears in the top-K ranked candidates, averaged over all queries. AR is the mean of R@K across K ∈ {1, 5, 10, 20}.

Input / output format

Input: An image and a text query (or vice versa), paired with a candidate pool of images/texts to rank.

Output: A ranked list of candidate images or texts based on cross-modal similarity scores.

Scoring recipe

def compute_recall_at_k(scores, gold_indices, k):
    hits = 0
    for s, gold in zip(scores, gold_indices):
        top_k = sorted(range(len(s)), key=lambda i: s[i], reverse=True)[:k]
        if any(g in top_k for g in gold):
            hits += 1
    return (hits / len(scores)) * 100

def compute_ar(recall_values):
    return sum(recall_values) / len(recall_values)

Common pitfalls

• The paper reports Recall values as percentages (e.g., 54.3), not decimals.
• Large candidate settings (Flickr30k-full/COCO-full) use the combined train+val+test sets as the retrieval pool, which differs from standard benchmark splits.
• Results are reported separately for image-to-text and text-to-image directions; AR is computed per direction, not averaged across directions unless specified.

Evidence (verbatim from paper)

We measure image-text retrieval by recall@$K$ (R@$K$) and the average R@$K$ (AR) for all $K$ for both image-to-text retrieval and text-to-image retrieval tasks. All metrics are the higher the better.

Citation

@misc{chen2022hivlp,
  title={HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2205.12105}
}

• arXiv: 2205.12105