Kosmos1 Eval

kosmos1-eval

Language Is Not All You Need: Aligning Perception with Language Models — Huang et al. (2023) (arXiv:2302.14045, 2023)

What this evaluates

Evaluates multimodal large language models on a comprehensive suite of perception-language, nonverbal reasoning, OCR-free text understanding, and web page comprehension tasks. It measures zero-shot and few-shot cross-modal transfer, in-context learning, and the ability to align visual perception with language generation without external tools or fine-tuning.

Datasets

• MS COCO Caption — total 123287; splits: train (113287), val (5000), test (5000)
• Flickr30k — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test-dev (-1)
• VizWiz — total ?; splits: test-dev (-1)
• Raven IQ Test — total 50; splits: test (50); repo https://aka.ms/kosmos-iq50
• Rendered SST-2 — total ?; splits: test (-1)
• HatefulMemes — total ?; splits: validation (-1)
• WebSRC — total ?; splits: test (-1)

Metrics

• CIDEr (primary) — range: [0, 100] (score)
  • Consensus-based Image Description Evaluation. Computes n-gram similarity between generated captions and multiple ground-truth captions, weighted by inverse document frequency (IDF) scores across a reference corpus.
• SPICE — range: [0, 1]
  • Semantic Propositional Image Caption Evaluation. Matches semantic propositions (objects, attributes, relationships) between prediction and ground truth using a scene graph parser, reporting F1 scores.
• VQA accuracy (primary) — range: [0, 100] (percent)
  • Percentage of correctly answered questions. Predictions are normalized per VQAv2 rules, and up to 3 correct answers are accepted per question.
• Accuracy — range: [0, 100] (percent)
  • Fraction of instances where the predicted candidate or label exactly matches the ground truth. Used for Raven IQ and Rendered SST-2.
• ROC AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring binary classification performance on HatefulMemes.
• Exact Match (EM) — range: [0, 100] (percent)
  • Fraction of predictions that exactly match the reference answer string character-for-character.
• F1 — range: [0, 100] (percent)
  • Harmonic mean of token-level precision and recall for open-ended answers.

Input / output format

Input: Varies by task: (1) Image + text prompt (e.g., 'An image of', 'Question: {question} Answer: {answer}'); (2) Flattened matrix images + verbal instruction for IQ test; (3) Image of rendered text or web page layout. Resolution fixed at 224x224 for vision tasks.

Output: Natural language caption, short answer string, or probability distribution over candidate images/labels. For IQ test, model outputs 'Yes'/'No' probability for each candidate appended separately.

Scoring recipe

def score_task(task, preds, gold, config):
    if task == 'captioning':
        cider = compute_cider(preds, gold)
        spice = compute_spice(preds, gold)
        return cider, spice
    elif task == 'vqa':
        preds = [p.split('</s>')[0] for p in preds]  # stop at EOS
        preds = [normalize(p) for p in preds]
        gold = [normalize(g) for g in gold]
        correct = sum(1 for p in preds if p in gold)
        return correct / len(preds)
    elif task == 'iq_test':
        probs = [model.predict_prob(cand, prompt) for cand in config['candidates']]
        pred = config['candidates'][argmax(probs)]
        return (pred == gold)
    elif task in ['sst2', 'hatefulmemes', 'websrc']:
        return compute_standard_metric(preds, gold, task)

Common pitfalls

• Using external OCR tools for OCR-free tasks violates the zero-shot setup and inflates performance.
• VQA answers must be truncated at the token before normalization and accuracy calculation.
• Few-shot settings sample demonstrations randomly from the training set, not the validation or test sets.
• Raven IQ evaluation appends each candidate separately and selects the one yielding the highest 'Yes' probability, rather than generating a direct answer.

Evidence (verbatim from paper)

We use COCOEvalCap to compute CIDEr and SPICE scores as the evaluation metrics. We follow the normalization rules of the VQAv2 evaluation code when computing the VQA accuracy. We evaluate the performance of VQA in an open-ended setting that Kosmos-1 generates answers and stops at the (“end of sequence”) token.

Citation

@misc{huang2023language,
  title={Language Is Not All You Need: Aligning Perception with Language Models},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2302.14045}
}

• arXiv: 2302.14045