# Fewmmbench Eval

> Evaluates multimodal large language models on few-shot learning capabilities across nine diverse tasks. It probes the models' ability to leverage in-context demonstrations (0, 4, or 8 shots) and chain-of-thought reasoning under controlled retrieval settings, measuring performance relative to zero-shot baselines. Use when the user wants to benchmark on FewMMBench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/fewmmbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/fewmmbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/fewmmbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/fewmmbench-eval

---


# fewmmbench-eval

> FewMMBench: A Benchmark for Multimodal Few-Shot Learning — Dogan et al. (2026) (arXiv:2602.21854, 2026)

## What this evaluates

Evaluates multimodal large language models on few-shot learning capabilities across nine diverse tasks. It probes the models' ability to leverage in-context demonstrations (0, 4, or 8 shots) and chain-of-thought reasoning under controlled retrieval settings, measuring performance relative to zero-shot baselines.

## Datasets

- **FewMMBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions based on ground-truth labels.

## Input / output format

**Input**: Image-text query with 2-4 candidate answers in a multiple-choice format, optionally augmented with few-shot demonstrations (0, 4, or 8 examples) retrieved via random or graph-cut-based methods, and optionally chain-of-thought reasoning prompts.

**Output**: Model generates a text response selecting one candidate answer. Predictions are normalized via regex-based post-processing or evaluated via perplexity-based pairwise ranking over candidates.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        normalized_pred = extract_option_regex(pred)
        if normalized_pred == gold:
            correct += 1
    return correct / len(gold_labels)
```

## Common pitfalls

- Regex-based post-processing may misinterpret open-ended or unexpected model outputs.
- Answer order bias can affect accuracy if models are sensitive to candidate placement.
- Perplexity-based ranking is an alternative that may yield different results than exact-match accuracy.

## Evidence (verbatim from paper)

> The primary metric is accuracy, computed as the proportion of correctly answered questions based on ground-truth labels. To ensure consistency in answer extraction, especially across models with varying output styles, we apply regex-based post-processing to normalize model predictions.

## Citation

```bibtex
@misc{dogan2026fewmmbench,
  title={FewMMBench: A Benchmark for Multimodal Few-Shot Learning},
  author={Dogan et al. (2026)},
  year={2026},
  note={arXiv:2602.21854}
}
```

- arXiv: 2602.21854