Marvel Eval

marvel-eval

MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning — Jiang et al. (2024) (arXiv:2404.13591, 2024)

What this evaluates

Evaluates multimodal large language models on multidimensional abstract visual reasoning and perceptual grounding. It probes the model's ability to recognize complex geometric and abstract patterns, track temporal/spatial changes, and perform multi-step visual reasoning across diverse puzzle configurations.

Datasets

• MARVEL — total 770; splits: test (770); repo https://github.com/1171-jpg/MARVEL_AVR

Metrics

• accuracy (primary) — range: [0, 1]
  • Extracted via regex matching from model responses (e.g., 'choice 4'). Failed extractions are re-extracted using GPT-4. Instance-based accuracy computes the fraction of correctly answered individual questions. Group-based accuracy requires the model to answer all questions within a predefined group correctly to receive a score of 1.

Input / output format

Input: Visual puzzle image(s) containing abstract/geometric shapes and multiple-choice options (typically 4 choices), presented in zero-shot or few-shot settings.

Output: Text response containing the selected choice (e.g., 'The correct answer is choice 4').

Scoring recipe

def extract_choice(response):
    match = re.search(r'choice\s+(\d+)', response)
    if not match:
        match = gpt4_extract(response) # fallback
    return match.group(1) if match else None

def instance_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

def group_accuracy(group_predictions, group_golds):
    correct_groups = 0
    for preds, golds in zip(group_predictions, group_golds):
        if all(p == g for p, g in zip(preds, golds)):
            correct_groups += 1
    return correct_groups / len(group_predictions)

Common pitfalls

• Group-based accuracy is an all-or-nothing metric per group; a single error in a group yields a score of 0, which heavily penalizes partial reasoning failures compared to standard per-instance accuracy.
• Models frequently fail at basic perceptual grounding (e.g., counting panels or identifying shapes), causing near-random performance on perception questions even when reasoning capabilities are theoretically present.
• Safety filters in some models (e.g., Gemini-pro-vision) refuse to answer a subset of puzzles (~7%), artificially deflating reported accuracy if not explicitly excluded from the denominator.

Evidence (verbatim from paper)

Following a similar setting as previous research evaluating MLLMs on the AVR benchmark (Ahrabian et al., 2024), we use regex matching to extract the choices picked (e.g., "choice $4''$ in the response "The correct answer is choice $4''$ ), with failure cases re-extracted by GPT-4 (Aher et al., 2023). We use accuracy as the metric, which is commonly used for evaluating multiple-choice questions and has been utilized by many AVR systems (Zhang et al., 2019; Hill et al., 2019). Based on the hierarchical evaluation framework, we evaluate MLLMs with two types of accuracy-based metrics:

1. Instance-based Accuracy considers questions separately. We report accuracy results for AVR question and fine-grained perception question.
2. Group-based Accuracy considers questions as group to assess the consistency in model reasoning (Jiang et al., 2023; Yuan et al., 2021). The model receives a score of 1 only if it correctly answers all questions within the same group. We report the group-based

accuracy result of combining all three coarse-grained perception questions and the further result after introducing fine-grained and AVR questions into the group.

Citation

@misc{jiang2024marvel,
  title={MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2404.13591}
}

• arXiv: 2404.13591