Vqa Captioning Eval

vqa-captioning-eval

OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text — Qingyun Li et al. (2024) (arXiv:2406.08418, 2024)

What this evaluates

Evaluates multimodal language models on visual question answering and image captioning tasks, probing their zero-shot and few-shot in-context learning capabilities with interleaved image-text inputs.

Datasets

• OKVQA — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• COCO — total ?; splits: val/test (-1)
• Flickr30k — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers for VQA tasks. Calculated as the number of exact matches between predicted and ground-truth answers divided by the total number of samples.
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation metric. Computes TF-IDF weighted n-gram similarity between generated captions and reference captions, averaging across n-grams of length 1 to 8.

Input / output format

Input: Interleaved sequence of visual and textual tokens. For few-shot evaluation, the input includes in-context examples sampled using RICES.

Output: Text string containing the predicted answer (for VQA) or generated caption (for image captioning).

Scoring recipe

def score_vqa(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return correct / len(gold)

def score_captioning(predictions, gold):
    # Use standard CIDEr implementation
    return compute_cider(predictions, gold)

Common pitfalls

• Few-shot in-context examples are sampled using RICES, not randomly, which significantly impacts reproducibility if not specified.
• Zero-shot evaluation for VQAv2 and VizWiz actually includes two text-only examples following the Flamingo protocol, so it is not pure zero-shot.
• The TextVQA prompt explicitly excludes OCR tokens, differing from standard OCR-augmented VQA evaluation settings.

Evidence (verbatim from paper)

The accuracy score is used for VQA, while CIDEr [108] is used for image captioning. Following OpenFlamingo [3], we extend the benchmarks to few-shot settings to assess in-context learning. Specifically, in-context examples are sampled using RICES [118].

Citation

@misc{li2024omnicorpus,
  title={OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text},
  author={Qingyun Li et al. (2024)},
  year={2024},
  note={arXiv:2406.08418}
}

• arXiv: 2406.08418