Crossmodal 3600 Eval

crossmodal-3600-eval

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset — Thapliyal et al. (2022) (arXiv:2205.12522, 2022)

What this evaluates

Evaluates multilingual image captioning models across 36 languages, probing their ability to generate stylistically coherent and culturally representative descriptions without relying on direct translation artifacts. It measures how well models generalize to low-resource and geographically diverse languages.

Datasets

• Crossmodal-3600 — total 3600; splits: test (3600)

Metrics

• CIDEr (primary) — range: [0, 1]
  • CIDEr (Consensus-based Image Description Evaluation) computes n-gram overlap between generated captions and reference captions, weighted by TF-IDF scores to downweight common words and upweight distinctive terms.

Input / output format

Input: An image and a target language code.

Output: A natural language caption in the specified target language.

Scoring recipe

def compute_cider(predictions, references):
    scores = []
    for pred, refs in zip(predictions, references):
        pred_ngrams = get_ngrams(pred, max_n=4)
        ref_ngrams = [get_ngrams(r, max_n=4) for r in refs]
        tfidf = compute_tfidf(ref_ngrams)
        sims = [cosine_similarity(pred_ngrams, ref_ngrams, tfidf) for ref_ngrams in ref_ngrams]
        scores.append(max(sims))
    return sum(scores) / len(scores)

Common pitfalls

• COCO-dev scores in this benchmark are computed using machine-translated references rather than human-written ones, which can artificially inflate CIDEr scores compared to standard English evaluations.
• CIDEr values can exceed 1.0; scores above 1 indicate strong n-gram overlap and are valid, not errors.
• Evaluations are reported per language, so averaging across languages without weighting by dataset size or language difficulty can mask performance disparities.

Evidence (verbatim from paper)

Table 8: CIDEr on XM3600 and COCO-dev for the best performing model BB+CC on all 35 languages. (COCO-dev computed using machine-translated references).

Citation

@misc{thapliyal2022crossmodal3600,
  title={Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset},
  author={Thapliyal et al. (2022)},
  year={2022},
  note={arXiv:2205.12522}
}

• arXiv: 2205.12522