Image Captioning Eval

image-captioning-eval

Deep Visual-Semantic Alignments for Generating Image Descriptions — Karpathy et al. (2014) (arXiv:1412.2306, 2014)

What this evaluates

Evaluates a model's ability to generate coherent natural language descriptions for images and align specific image regions with corresponding text segments. It measures both retrieval quality and generation fidelity against human-written references.

Datasets

• Flickr8K — total 8000; splits: train (-1), val (1000), test (1000)
• Flickr30K — total 31000; splits: train (-1), val (1000), test (1000)
• MSCOCO — total 123000; splits: train (-1), val (5000), test (5000)

Metrics

• BLEU (primary) — range: percent
  • Computes n-gram precision (up to 4-grams) with a brevity penalty against up to 5 human reference sentences per image.
• METEOR — range: percent
  • Measures alignment between candidate and reference sentences using synonymy, stemming, and exact matches, normalized against 5 references.
• CIDEr — range: percent
  • Computes TF-IDF weighted n-gram similarity between candidate and reference sentences, normalized against 5 references.
• Recall@K — range: [0, 1]
  • Measures the fraction of times a correct image or sentence is found among the top K results in a ranked retrieval list.
• Median Rank — range: rank
  • Reports the median position of the ground truth item in the sorted list of retrieved candidates.

Input / output format

Input: Image (full frame or cropped region) and optionally a reference sentence for alignment tasks.

Output: Generated natural language sentence (for captioning) or ranked list of image-sentence pairs with alignment scores (for alignment).

Scoring recipe

def compute_generation_metrics(predictions, references):
    # references is a list of 5 human sentences per image
    bleu = compute_bleu(predictions, references, max_n=4)
    meteor = compute_meteor(predictions, references)
    cider = compute_cider(predictions, references)
    return bleu, meteor, cider

def compute_alignment_metrics(scores, ground_truth_indices):
    ranked_indices = argsort(scores, descending=True)
    median_rank = median([ranked_indices.tolist().index(gt) + 1 for gt in ground_truth_indices])
    recall_k = mean([1 if gt in ranked_indices[:K] else 0 for gt in ground_truth_indices])
    return median_rank, recall_k

Common pitfalls

• Using fewer than 5 reference sentences per image violates the coco-caption evaluation standard used in the paper.
• Confusing full-image captioning results with region-level captioning results, as the region model is evaluated on a separate 200-image AMT-annotated test split.
• Comparing BLEU scores across models without accounting for sentence length differences, as the paper notes longer sentences can artificially lower BLEU despite higher semantic accuracy.

Evidence (verbatim from paper)

We report the BLEU, METEOR and CIDEr scores computed with the coco-caption code 222https://github.com/tylin/coco-caption. Each method evaluates a candidate sentence by measuring how well it matches a set of five reference sentences written by humans.

Citation

@misc{karpathy2014deep,
  title={Deep Visual-Semantic Alignments for Generating Image Descriptions},
  author={Karpathy et al. (2014)},
  year={2014},
  note={arXiv:1412.2306}
}

• arXiv: 1412.2306