qhjqhj00 nllb-clip-retrieval-eval
NLLB-CLIP -- train performant multilingual image retrieval model on a budget — Visheratin (2023) (arXiv:2309.01859, 2023)
What this evaluates
Probes multilingual image-text retrieval capability across low-resource languages. It evaluates how effectively the model aligns visual and textual representations when trained with limited data and frozen encoders.
Datasets
- XTD200 — total 106246; splits: test (-1)
- Flickr30k-200 — total ?; splits: test (-1)
Metrics
R@10(primary) — range: [0, 1]- Recall@10 measures the fraction of queries for which the ground-truth matching item appears within the top 10 retrieved results.
Input / output format
Input: Image paired with a text caption in one of 201 languages.
Output: Retrieval ranking or similarity score between the image and text caption.
Scoring recipe
def recall_at_k(retrieved_indices, ground_truth_index, k=10):
return 1.0 if ground_truth_index in retrieved_indices[:k] else 0.0
# Average over all queries in the dataset
total_recall = sum(recall_at_k(preds[i], gold[i]) for i in range(len(preds)))
metric_value = total_recall / len(preds)
Common pitfalls
- Training the full model (unfreezing the image encoder) often fails to converge or underperforms compared to freezing the image encoder.
- Using larger text encoders can degrade performance due to insufficient data to align them with the image encoder.
- Training only the projection layers can surprisingly outperform full fine-tuning.
Evidence (verbatim from paper)
The smallest model variants (e.g., b-b and l-b) could not converge to get R@10 higher than 20% when we performed training of the full model.
Citation
@misc{visheratin2023nllbclip,
title={NLLB-CLIP -- train performant multilingual image retrieval model on a budget},
author={Visheratin (2023)},
year={2023},
note={arXiv:2309.01859}
}
- arXiv: 2309.01859