Magicmirror Eval

magicmirror-eval

MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation — Wang et al. (2025) (arXiv:2509.10260, 2025)

What this evaluates

Evaluates text-to-image generation models on their ability to produce images free of fine-grained artifacts, specifically probing subject anatomy, attributes, and interactions. It measures detection accuracy using a hierarchical taxonomy of artifact types to benchmark model robustness against visual inconsistencies.

Datasets

• MagicData340K — total 340000; splits: train (-1)

Metrics

• F1-Score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (P * R) / (P + R). Calculated per class and macro/micro averaged across L2 categories.
• Precision — range: [0, 1]
  • Ratio of correctly identified artifact instances to all instances predicted as artifacts.
• Recall — range: [0, 1]
  • Ratio of correctly identified artifact instances to all actual artifact instances.
• Overall Score — range: [0, 100]
  • Aggregated benchmark score across interaction, human, animal, and object categories for MagicBench evaluation.

Input / output format

Input: A text prompt and the corresponding generated image.

Output: Binary classification label (artifact present/absent), hierarchical L2 category labels, and a Chain-of-Thought inspection explanation.

Scoring recipe

def calculate_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• Larger VLMs often perform worse on artifact detection due to increased conservatism, leading to significantly lower recall.
• Interaction and object morphology categories are inherently harder; models struggle to distinguish actual artifacts from low-quality image regions or diverse object types.
• Without multi-level reward consistency, VLMs may produce disorganized explanations and suffer from reward hacking during GRPO training.

Evidence (verbatim from paper)

We evaluate model performance using Precision, Recall, and F1-Score, calculated for both the overall artifact detection task and for each of our primary L2 categories.

Citation

@misc{wang2025magicmirror,
  title={MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.10260}
}

• arXiv: 2509.10260