Carpe Eval

carpe-eval

CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models — Lee et al. (2026) (arXiv:2601.13622, 2026)

What this evaluates

Evaluates the visual classification and vision-language understanding capabilities of large vision-language models (LVLMs) under context-aware ensemble prompting. It probes fine-grained visual recognition, scientific question answering, text-rich VQA, hallucination detection, and multimodal reasoning across diverse benchmarks.

Datasets

• ImageNet — total ?; splits: test (-1)
• Caltech101 — total ?; splits: test (-1)
• Flower102 — total ?; splits: test (-1)
• Food101 — total ?; splits: test (-1)
• ScienceQA (image subset) — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• CV-Bench — total ?; splits: test (-1)
• MMVP — total ?; splits: test (-1)

Metrics

• accuracy / F1 score / scaled MME score (primary) — range: [0, 100] percent
  • Standard classification accuracy (correct predictions divided by total instances). For POPE, the F1 score is reported. For MME, raw scores are scaled to a 0–100 range before averaging across benchmarks. All other VL benchmarks report standard accuracy.

Input / output format

Input: Paired image and text inputs. For classification, images with standardized prompt templates (e.g., 'Identify the object in this image:'). For VL tasks, images with questions or instruction-following prompts.

Output: Textual predictions: class labels for classification, direct answers for VQA/QA, and yes/no/maybe responses for hallucination detection (POPE).

Scoring recipe

def compute_metric(predictions, golds, dataset_name):
    if dataset_name == 'POPE':
        tp = sum(1 for p, g in zip(predictions, golds) if p == g == 'yes')
        fp = sum(1 for p, g in zip(predictions, golds) if p == 'yes' and g == 'no')
        fn = sum(1 for p, g in zip(predictions, golds) if p == 'no' and g == 'yes')
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif dataset_name == 'MME':
        return (sum(predictions) / len(predictions)) * 100
    else:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds) * 100

Common pitfalls

• MME scores must be scaled to 100 for cross-benchmark averaging, not used as raw values.
• POPE reports F1 score rather than accuracy, requiring careful precision/recall calculation.
• ScienceQA evaluation strictly uses the image subset, not the full multimodal dataset.
• ImageNet prompting uses 20 different templates with a 50/50 split between open- and closed-world formats to prevent overfitting.

Evidence (verbatim from paper)

MME scores are scaled to 100 for averaging; SQA refers to the image subset of ScienceQA; POPE is reported with F1 score; all others are accuracy.

Citation

@misc{lee2026carpe,
  title={CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2601.13622}
}

• arXiv: 2601.13622