Mscoco Eval

mscoco-eval

Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

What this evaluates

Evaluates image captioning quality by scoring generated captions against crowdworker references and expert THumB 1.0 scores. It measures how well automatic metrics correlate with human judgments across different captioning systems.

Datasets

• MSCOCO — total ?; splits: test (-1)

Metrics

• RefCLIP-S (primary) — range: [0, 1]
  • A reference-based metric that uses CLIP image and text features to compute similarity between image and caption.
• RefOnlyC — range: [0, 1]
  • A variant of RefCLIP-S that removes image features and only uses reference text features from CLIP.
• CIDEr — range: [0, 1]
  • A reference-based metric that computes TF-IDF weighted n-gram similarity between hypothesis and references.

Input / output format

Input: Image and optionally reference captions.

Output: Generated caption.

Scoring recipe

scores = []
for img, refs, caption in test_images:
    metric_score = compute_metric(caption, refs, img)
    human_score = average(expert_THumB_scores[caption])
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])

Common pitfalls

• Using RefCLIP-S instead of RefOnlyC when isolating text-only evaluation quality, as RefCLIP-S incorporates image features.
• Not following the standard Karpathy split for train/test partitioning.

Evidence (verbatim from paper)

Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: RefCLIP-S ... Ensemble of Metrics: 2.08·RefCLIP-S+1.51·RefOnlyC+0.82·CIDEr ... We use the MSCOCO dataset... We apply the standard Karpathy split. For each of 500 test images, rubric-based evaluations (THumB 1.0) are available for five systems... we use all five crowdworker captions as a reference set for reference-based metrics.

Citation

@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}

• arXiv: 2112.04139