Visual Commonsense Eval

visual-commonsense-eval

Improving Visual Commonsense in Language Models via Multiple Image Generation — Yariv et al. (2024) (arXiv:2406.13621, 2024)

What this evaluates

Evaluates language models' zero-shot visual and textual commonsense reasoning without relying on ground-truth images. It probes the model's ability to infer object properties (color, shape, size) and answer general knowledge questions by internally generating and fusing multiple image variations from text prompts.

Datasets

• ImageNetVC — total 4076; splits: test (-1)
• Object Commonsense (Memory Color, Color Terms, ViComTe, Size) — total ?; splits: test (-1)
• Commonsense Reasoning (PIQA, SIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, CommonsenseQA) — total ?; splits: test (-1)
• Reading Comprehension (BoolQ, SQuAD 2.0, QuAC) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: fraction of correctly predicted answers across all evaluation instances. Used for multiple-choice and zero-shot QA tasks.
• exact match (EM) — range: [0, 1]
  • Binary score: 1 if the predicted answer exactly matches the gold answer string (case-insensitive, normalized whitespace), 0 otherwise. Used for open-ended QA datasets.

Input / output format

Input: Text-only questions or prompts. The model internally generates multiple images from the text and fuses their prediction probabilities, but the external input contains no visual data.

Output: Predicted answer token(s) or selected option from a predefined set (e.g., 'yellow', 'yes', 'no', or a multiple-choice letter).

Scoring recipe

def compute_metrics(predictions, golds, dataset_name):
    if dataset_name in ['SQuAD 2.0', 'QuAC']:
        return sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g)) / len(golds)
    elif dataset_name == 'BoolQ':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    else:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

Common pitfalls

• Zero-shot evaluation: models are not fine-tuned on any evaluation split; performance relies entirely on pre-training and the proposed image-generation fusion.
• Strict prompt adherence required: Object Commonsense tasks must follow Wang et al. (2023) formatting, and Reading Comprehension must follow Ouyang et al. (2022) settings to match reported baselines.
• Internal image generation is not exposed during inference: the model generates images internally and fuses probabilities, so external evaluators only see text prompts and text outputs.

Evidence (verbatim from paper)

We evaluate SQuAD and QuAC using the settings recommended by Ouyang et al. ([2022]) and report the exact match (EM) score. For BoolQ, we consider a zero-shot binary setup by selecting the highest probability between the yes and no tokens.

Citation

@misc{yariv2024improvingvisualcommonsense,
  title={Improving Visual Commonsense in Language Models via Multiple Image Generation},
  author={Yariv et al. (2024)},
  year={2024},
  note={arXiv:2406.13621}
}

• arXiv: 2406.13621