Multimodal Instruction Eval

multimodal-instruction-eval

Concepts or Skills? Rethinking Instruction Selection for Multi-modal Models — Bai et al. (2025) (arXiv:2508.10339, 2025)

What this evaluates

This protocol evaluates how concept- versus skill-targeted instruction selection strategies improve vision-language model performance under strict data budget constraints. It probes the model's zero-shot generalization across diverse tasks including VQA, OCR, spatial reasoning, and scientific understanding by aligning training data with the benchmark's dominant cognitive demand.

Datasets

• VQAv2 — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• ScienceQA (SQA-I) — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMBench (en) — total ?; splits: test (-1)
• LLaVA-Bench — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• OK-VQA — total ?; splits: test (-1)
• ST-VQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Proportion of correctly predicted answers out of total questions. Calculated as the number of instances where the model's normalized output exactly matches the ground truth answer divided by the total number of instances.
• exact match — range: [0, 1]
  • Binary scoring metric where the model's generated text string must exactly match the ground truth answer string after case-insensitive normalization and whitespace stripping.

Input / output format

Input: A single image paired with a natural language question or instruction.

Output: A natural language text response or selected option corresponding to the question.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for p, g in zip(predictions, golds):
        if normalize(p) == normalize(g):
            correct += 1
    return correct / len(golds)

def compute_exact_match(predictions, golds):
    matches = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return matches / len(golds)

Common pitfalls

• Different benchmarks use different evaluation metrics (accuracy vs exact match vs MME aggregate score), making direct cross-benchmark comparison of raw scores invalid without normalization.
• Only the random baseline is evaluated over three seeds; targeted baselines use single runs due to compute constraints, limiting statistical significance assessment for targeted methods.
• MME and LLaVA-Bench rely on complex or proprietary scoring rubrics not detailed in the text, hindering exact reproduction.

Evidence (verbatim from paper)

We evaluate each model in a zero-shot setting on the downstream benchmarks, reporting task-specific metrics such as accuracy (GQA, ScienceQA) and exact match (TextVQA, OCR-VQA) where applicable. The random baseline is repeated with three random seeds to account for training variability, and we report the average performance along with standard deviations.

Citation

@misc{bai2025concepts,
  title={Concepts or Skills? Rethinking Instruction Selection for Multi-modal Models},
  author={Bai et al. (2025)},
  year={2025},
  note={arXiv:2508.10339}
}

• arXiv: 2508.10339