Image Captioning Retrieval Eval

image-captioning-retrieval-eval

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts — Changpinyo et al. (2021) (arXiv:2102.08981, 2021)

What this evaluates

Evaluates vision-language models on image captioning and image-text retrieval tasks to measure zero-shot and fine-tuned generalization on long-tail visual concepts and out-of-domain data.

Datasets

• nocaps — total ?; splits: test (-1)
• COCO Captions — total ?; splits: val2017 (-1)
• Flickr30K — total ?; splits: test (-1)
• LocNar Flickr30K — total ?; splits: test (-1)

Metrics

• CIDEr (primary) — range: other
  • Computes n-gram TF-IDF weighted cosine similarity between predicted and ground-truth captions against a reference corpus. Higher scores indicate better semantic and lexical alignment.
• BLEU-4 — range: [0, 1]
  • Geometric mean of modified precision scores for 1- to 4-grams, with brevity penalty to discourage overly short outputs.
• SPICE — range: [0, 1]
  • Measures semantic proposition overlap between predicted and ground-truth captions using scene graph parsing.
• R1 — range: [0, 1]
  • Fraction of queries where the ground-truth match appears in the top-1 ranked results.
• R5 — range: [0, 1]
  • Fraction of queries where the ground-truth match appears in the top-5 ranked results.
• R10 — range: [0, 1]
  • Fraction of queries where the ground-truth match appears in the top-10 ranked results.

Input / output format

Input: Image file and optional text prompt/context for captioning; image-text pairs for retrieval.

Output: Natural language caption string for each image; ranked list of matching texts/images for retrieval.

Scoring recipe

def compute_cider(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        scores.append(cider_metric.compute([pred], [gold]))
    return sum(scores) / len(scores)

def compute_recall_at_k(predictions, golds, k):
    hits = sum(1 for pred_list, gold in zip(predictions, golds) if gold in pred_list[:k])
    return hits / len(predictions)

Common pitfalls

• Zero-shot captioning scores are artificially low because models lack the COCO captioning style; fine-tuning is required to match automatic metric expectations.
• Over-fine-tuning on in-domain data (COCO) degrades out-of-domain generalization, causing a trade-off between in-domain metrics and novel concept recognition.
• BLEU-4 and SPICE are highly sensitive to exact lexical overlap, which may not reflect semantic alignment for long-tail concepts.

Evidence (verbatim from paper)

With a fine-tuned model, the benefit of transfer learning using pre-training on this task is clear (Row 1 vs. Rows 4,5,6), with CC12M outperforming CC3M by +14.2 CIDEr points and another +2.8 with CC3M+CC12M.

Citation

@misc{changpinyo2021conceptual12m,
  title={Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts},
  author={Changpinyo et al. (2021)},
  year={2021},
  note={arXiv:2102.08981}
}

• arXiv: 2102.08981