Multilingual Clip Eval

multilingual-clip-eval

Evaluation of Multilingual Image Captioning: How far can we get with CLIP models? — Gonçalo Gomes, Chrysoula Zerva, Bruno Martins (2025) (arXiv:2502.06600, 2025)

What this evaluates

This benchmark evaluates how well CLIP-based models can assess the quality and semantic alignment of image captions across multiple languages. It measures the correlation between automated CLIPScore metrics and human quality judgments, as well as classification accuracy on foil-caption tasks.

Datasets

• Flickr8K-Expert — total 5664; splits: test (5664)
• Flickr8K-CF — total 47830; splits: test (47830)
• Composite — total 13146; splits: test (13146)
• VICR — total 15646; splits: train (10175), val (2310), test (3161)
• VALSE — total 13408; splits: test (13408)
• XVNLI — total ?; splits: test (-1)
• MaRVL — total ?; splits: test (-1)

Metrics

• Spearman ρ (primary) — range: percent
  • Rank correlation coefficient between predicted CLIPScore values and human quality ratings. Reported as a percentage (e.g., 82.4 corresponds to 0.824).
• Kendall τ_b / τ_c — range: percent
  • Rank correlation coefficient handling ties (τ_b) or not (τ_c) between predicted scores and human ratings. Reported as a percentage.
• Accuracy — range: percent
  • Proportion of correctly classified image-caption pairs in binary foil-caption tasks (entailed vs contradiction).

Input / output format

Input: Image and caption pair (optionally with a reference caption for reference-based variants).

Output: A continuous scalar score representing the image-caption alignment/quality.

Scoring recipe

scores = [clip_model(image, caption) for image, caption in dataset]
correlation = spearmanr(scores, human_ratings) * 100
# or kendalltau(scores, human_ratings) * 100
# For classification: accuracy = mean(pred_label == gold_label) * 100
# Note: 1,000 iterations of stratified 80% sampling used for statistical significance.

Common pitfalls

• Correlation metrics are reported as percentages (e.g., 82.4) rather than decimals, which may confuse readers expecting a [-1, 1] range.
• Multilingual evaluation relies on machine-translated English benchmarks; translation quality directly impacts the validity of human judgments across target languages.
• Statistical significance is assessed via 1,000 iterations of stratified random sampling (80% subsets) rather than a single train/test split.

Evidence (verbatim from paper)

We evaluate the different models using correlation with human judgements, and also through classification tasks. Regarding the correlation experiments, we measure performance using three different correlation coefficients, namely Spearman ρ and Kendall τ with variations b and c. The correlation metrics are formally defined in Appendix[B]. For the classification experiments, we measure accuracy under the assumption that a caption entailed by an image should reflect a higher CLIPScore than a contradiction/foil caption.

Citation

@misc{gomes2025evaluationmultilingual,
  title={Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?},
  author={Gonçalo Gomes, Chrysoula Zerva, Bruno Martins (2025)},
  year={2025},
  note={arXiv:2502.06600}
}

• arXiv: 2502.06600