Countqa Eval

countqa-eval

CountQA: How Well Do MLLMs Count in the Wild? — Tamarapalli et al. (2025) (arXiv:2508.06585, 2025)

What this evaluates

This benchmark evaluates the object counting and spatial individuation capabilities of multimodal large language models (MLLMs) on real-world images characterized by high density, clutter, and occlusion. It probes whether generalist models can perform precise, fine-grained visual grounding and numerical reasoning out-of-the-box without specialized training.

Datasets

• CountQA — total 1528; splits: test (1528)

Metrics

• Exact Match (EM) (primary) — range: percent
  • Percentage of predictions where the final extracted number is identical to the ground truth integer.
• Relaxed Accuracy@5% (RA@5%) — range: percent
  • Percentage of predictions where the predicted count falls within 5% of the ground truth value.
• Relaxed Accuracy@10% (RA@10%) — range: percent
  • Percentage of predictions where the predicted count falls within 10% of the ground truth value.

Input / output format

Input: An image paired with a natural language question asking for the count of specific objects. Models are prompted with a system instruction to act as a counting assistant and return only a single integer.

Output: A single integer representing the predicted object count.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    em_count = 0
    ra5_count = 0
    ra10_count = 0
    for pred, gt in zip(predictions, ground_truths):
        if pred == gt:
            em_count += 1
        if abs(pred - gt) <= 0.05 * gt:
            ra5_count += 1
        if abs(pred - gt) <= 0.10 * gt:
            ra10_count += 1
    n = len(predictions)
    return {
        'Exact Match (EM)': em_count / n,
        'Relaxed Accuracy@5%': ra5_count / n,
        'Relaxed Accuracy@10%': ra10_count / n
    }

Common pitfalls

• Models frequently fail to adhere to the strict single-integer output format, producing verbose text that requires external parsing (e.g., using an LLM) to extract the count.
• Counting performance degrades sharply as the number of objects increases, exposing a fundamental gap in numerical and spatial reasoning capabilities.
• High-density clutter and occlusion significantly impact accuracy, revealing deficits in fine-grained visual grounding.

Evidence (verbatim from paper)

The primary metric for our evaluation is Exact Match (EM), which measures the percentage of predictions where the final extracted number is identical to the ground truth integer. To provide a more nuanced view of performance, especially on questions with large counts, we also report on two Relaxed Accuracy (RA) metrics. An answer is considered correct under RA@5% or RA@10% if the predicted count falls within 5% or 10% of the ground truth value, respectively.

Citation

@misc{tamarapalli2025countqa,
  title={CountQA: How Well Do MLLMs Count in the Wild?},
  author={Tamarapalli et al. (2025)},
  year={2025},
  note={arXiv:2508.06585}
}

• arXiv: 2508.06585