Truemicl Eval

truemicl-eval

True Multimodal In-Context Learning Needs Attention to the Visual Context — Shuo Chen et al. (2025) (arXiv:2507.15807, 2025)

What this evaluates

This benchmark evaluates a model's ability to perform true multimodal in-context learning by requiring it to solve tasks that depend on both visual and textual information from provided demonstrations. It probes whether models can correctly attend to and utilize visual context in few-shot examples rather than relying on superficial textual patterns or prior knowledge.

Datasets

• TrueMICL — total ?; splits: test (-1), support (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered queries. For numerical answers, strict exact match is required. For textual answers, keyword-based matching is used to determine correctness.

Input / output format

Input: A query image, a text prompt, and 4 demonstration examples (image-text pairs) retrieved from a support set.

Output: A textual or numerical answer string generated by the model.

Scoring recipe

def compute_accuracy(predictions, golds, is_numerical):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if is_numerical:
            if pred.strip() == gold.strip():
                correct += 1
        else:
            keywords = gold.split() # Simplified keyword extraction per paper
            if any(kw.lower() in pred.lower() for kw in keywords):
                correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Models often ignore visual tokens in demonstrations and rely solely on textual patterns, leading to artificially high accuracy on standard benchmarks but failure on TrueMICL.
• Keyword-based matching for textual answers lacks a standardized threshold or explicit keyword list in the paper, which may cause inconsistent evaluation across different implementations.
• Using standard VL datasets (e.g., VQAv2) for MICL evaluation is misleading because they do not require visual context from demonstrations to solve correctly.

Evidence (verbatim from paper)

The evaluation metric for TrueMICL is accuracy. For numerical answers, only strict matching is considered correct, while textual answers are evaluated through keyword-based matching.

Citation

@misc{chen2025truemicl,
  title={True Multimodal In-Context Learning Needs Attention to the Visual Context},
  author={Shuo Chen et al. (2025)},
  year={2025},
  note={arXiv:2507.15807}
}

• arXiv: 2507.15807