Fewmmbench Eval

fewmmbench-eval

FewMMBench: A Benchmark for Multimodal Few-Shot Learning — Dogan et al. (2026) (arXiv:2602.21854, 2026)

What this evaluates

Evaluates multimodal large language models on few-shot learning capabilities across nine diverse tasks. It probes the models' ability to leverage in-context demonstrations (0, 4, or 8 shots) and chain-of-thought reasoning under controlled retrieval settings, measuring performance relative to zero-shot baselines.

Datasets

• FewMMBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered questions based on ground-truth labels.

Input / output format

Input: Image-text query with 2-4 candidate answers in a multiple-choice format, optionally augmented with few-shot demonstrations (0, 4, or 8 examples) retrieved via random or graph-cut-based methods, and optionally chain-of-thought reasoning prompts.

Output: Model generates a text response selecting one candidate answer. Predictions are normalized via regex-based post-processing or evaluated via perplexity-based pairwise ranking over candidates.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        normalized_pred = extract_option_regex(pred)
        if normalized_pred == gold:
            correct += 1
    return correct / len(gold_labels)

Common pitfalls

• Regex-based post-processing may misinterpret open-ended or unexpected model outputs.
• Answer order bias can affect accuracy if models are sensitive to candidate placement.
• Perplexity-based ranking is an alternative that may yield different results than exact-match accuracy.

Evidence (verbatim from paper)

The primary metric is accuracy, computed as the proportion of correctly answered questions based on ground-truth labels. To ensure consistency in answer extraction, especially across models with varying output styles, we apply regex-based post-processing to normalize model predictions.

Citation

@misc{dogan2026fewmmbench,
  title={FewMMBench: A Benchmark for Multimodal Few-Shot Learning},
  author={Dogan et al. (2026)},
  year={2026},
  note={arXiv:2602.21854}
}

• arXiv: 2602.21854