Coco Caption Re Ranking Eval

coco-caption-re-ranking-eval

Visual Semantic Relatedness Dataset for Image Captioning — Sabir et al. (2023) (arXiv:2301.08784, 2023)

What this evaluates

Evaluates the ability of vision-language models to re-rank candidate image captions based on their semantic alignment with extracted visual context. It probes how well models can leverage object-level visual information to improve caption relevance and accuracy.

Datasets

• COCO Captions (Karpathy test split) — total ?; splits: test (-1); repo https://github.com/ahmedssabir/Textual-Visual-Semantic-Dataset

Metrics

• BERTScore (B-S) (primary) — range: [0, 1]
  • Computes the mean cosine similarity between BERT embeddings of the predicted caption and the ground-truth reference caption.
• BLEU-4 — range: [0, 1]
  • Measures n-gram precision up to 4-grams between predicted and reference captions, with a brevity penalty.
• METEOR — range: [0, 1]
  • Aligns predicted and reference captions using exact, stem, synonym, and paraphrase matches, then computes a weighted harmonic mean with a penalty for fragmentation.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence between predicted and reference captions to capture sentence-level structure.
• CIDEr — range: [0, 1]
  • Computes TF-IDF weighted n-gram similarity between predicted and reference captions, emphasizing rare but informative words.
• SPICE — range: [0, 1]
  • Evaluates caption quality by matching scene graphs (objects, attributes, relationships) between predicted and reference captions.

Input / output format

Input: Candidate image captions (typically top-9 from beam search) paired with extracted visual context (top-3 object contexts or overlapping object-caption terms).

Output: Re-ranked list of candidate captions, or a binary/continuous semantic relatedness score [0,1] used for filtering and ranking.

Scoring recipe

def evaluate(gold_captions, pred_captions):
    # Standard COCO metrics
    bleu4 = coco_bleu4(gold_captions, pred_captions)
    meteor = coco_meteor(gold_captions, pred_captions)
    rouge_l = coco_rouge(gold_captions, pred_captions)
    cider = coco_cider(gold_captions, pred_captions)
    spice = coco_spice(gold_captions, pred_captions)
    # BERTScore
    pred_embeds = bert_model.encode(pred_captions)
    gold_embeds = bert_model.encode(gold_captions)
    b_score = cosine_similarity(pred_embeds, gold_embeds).mean()
    return {'BLEU-4': bleu4, 'METEOR': meteor, 'ROUGE-L': rouge_l, 'CIDEr': cider, 'SPICE': spice, 'BERTScore': b_score}

Common pitfalls

• The re-ranking pipeline applies a similarity threshold (e.g., ≥ 0.2, 0.3, 0.4) to filter captions before ranking, which drastically changes final metric scores and is not always reported consistently.
• Visual context is extracted using external off-the-shelf classifiers (ResNet-152, CLIP, Faster R-CNN), so evaluation results conflate the re-ranker's performance with the detector's accuracy.
• Human evaluation is limited to 19 images and 12 subjects, yielding only a ~50% agreement rate that does not statistically validate the re-ranking improvements.

Evidence (verbatim from paper)

Evaluation Metric. We use the official COCO offline evaluation suite, producing several widely used caption quality metrics: BLEU [34] METEOR [4], ROUGE [29], CIDEr [41], SPICE [2] and the semantic-similarity based metric BERTScore (B-S) [43].

Citation

@misc{sabir2023visualsemantic,
  title={Visual Semantic Relatedness Dataset for Image Captioning},
  author={Sabir et al. (2023)},
  year={2023},
  note={arXiv:2301.08784}
}

• arXiv: 2301.08784