Text Image Retrieval Eval

text-image-retrieval-eval

15M Multimodal Facial Image-Text Dataset — Dawei Dai et al. (2024) (arXiv:2407.08515, 2024)

What this evaluates

Evaluates the model's ability to align facial images with their textual descriptions by retrieving the correct image given a text query, and vice versa. It measures how well the model learns cross-modal semantic correspondence for face-centric data.

Datasets

• CelebA-Caption — total 182732; splits: train (162770), test (19962)
• MM-CelebA — total 30000; splits: train (24000), test (6000)

Metrics

• R@5, R@10 (primary) — range: percent
  • Recall at top-K, measuring the percentage of queries where the ground-truth image (or text) appears in the top-K retrieved results. Evaluated in both Text->Image and Image->Text directions.

Input / output format

Input: A facial image and its corresponding caption text (or vice versa) used as a query against a candidate pool.

Output: A ranked list of retrieved images or texts from the candidate pool.

Scoring recipe

def compute_rk(predictions, gold, k):
    ranks = []
    for pred_list, gold_item in zip(predictions, gold):
        if gold_item in pred_list:
            ranks.append(pred_list.index(gold_item) + 1)
        else:
            ranks.append(k + 1)
    return sum(1 for r in ranks if r <= k) / len(ranks) * 100

Common pitfalls

• Freezing backbone weights and only fine-tuning linear layers may limit performance compared to full fine-tuning baselines.
• Retrieval is evaluated in both directions (Text->Image and Image->Text), which must be reported separately and are not interchangeable.

Evidence (verbatim from paper)

We adopted the Top-K retrieval index, where R@5 and R@10 represent the top-5 and top-10 accuracy, respectively.

Citation

@misc{dai2024facecaption,
  title={15M Multimodal Facial Image-Text Dataset},
  author={Dawei Dai et al. (2024)},
  year={2024},
  note={arXiv:2407.08515}
}

• arXiv: 2407.08515