Mmad Eval

mmad-eval

MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection — Jiang et al. (2024) (arXiv:2410.09453, 2024)

What this evaluates

Evaluates Multimodal Large Language Models (MLLMs) on industrial anomaly detection tasks, probing their ability to perform fine-grained visual reasoning, multi-image comparison, and defect-related classification, localization, and description. It specifically tests whether models can leverage template normal images and domain knowledge to identify and analyze anomalies in industrial products.

Datasets

• MMAD — total 39672; splits: test (39672); repo https://github.com/jam-cc/MMAD

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. For the anomaly discrimination subtask, accuracy is computed separately for normal and abnormal samples, then averaged. The overall average accuracy is the mean across all seven subtasks.

Input / output format

Input: A query image, a multiple-choice question, and randomized answer options. Optionally, a single normal template image is provided (1-shot or 1-shot+ setting) to aid comparison.

Output: A single letter or text corresponding to the selected multiple-choice option. If the model outputs free text, it is matched to the closest option.

Scoring recipe

def compute_accuracy(predictions, golds, subtask):
    if subtask == 'anomaly_discrimination':
        normal_acc = mean([p == g for p, g in zip(predictions, golds) if gold == 'normal'])
        abnormal_acc = mean([p == g for p, g in zip(predictions, golds) if gold == 'abnormal'])
        return (normal_acc + abnormal_acc) / 2
    else:
        return mean([p == g for p, g in zip(predictions, golds)])
# Final score is the average of subtask accuracies

Common pitfalls

• Option letters and order are randomized per instance to prevent positional bias, which can cause models relying on fixed prompts to fail.
• The anomaly discrimination subtask requires calculating accuracy separately for normal and abnormal samples before averaging, rather than using overall accuracy due to class imbalance.
• Many open-source MLLMs default to single-image input; enabling the 1-shot+ template setting requires framework modifications or specific multi-image handling.
• Models often output free-text answers instead of option letters, requiring a closest-match string matching step to score correctly.

Evidence (verbatim from paper)

We will randomize the letters and order of the options and use the accuracy of responses as a metric. If the model does not provide any option, we will automatically match the closest option to the output as the answer. It is worth noting that, in the anomaly discrimination subtask, due to the imbalance distribution, we will separately calculate the accuracy of normal and abnormal samples and then use their mean as the final accuracy.

Citation

@misc{jiang2024mmad,
  title={MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2410.09453}
}

• arXiv: 2410.09453