Genimage Eval

genimage-eval

Explainable Synthetic Image Detection through Diffusion Timestep Ensembling — Wu et al. (2025) (arXiv:2503.06201, 2025)

What this evaluates

This benchmark probes a model's ability to distinguish real from AI-generated images across multiple diffusion and GAN generators, and to identify specific synthetic flaws in generated images. It evaluates standard detection accuracy, cross-generator generalization, and robustness to common image perturbations like blur, rotation, and brightness shifts.

Datasets

• GenImage — total 1000000; splits: train (-1), test (-1)
• GENHARD — total ?; splits: test (-1)
• GENEXPLAIN — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Standard classification accuracy calculated as the proportion of correctly predicted labels out of total instances. Reported separately for hard samples (prior number) and original samples (posterior number) in the main results table.
• Exact Match (EM) accuracy — range: percent
  • Proportion of instances where all 14 flaw-type labels are predicted correctly.
• Mean Average Precision (mAP) — range: [0, 1]
  • Mean of the average precision scores computed independently across all 14 flaw classification labels.

Input / output format

Input: RGB images (256×256 or 512×512) processed through DDIM inversion to extract intermediate noised states. Features are extracted using CLIP ViT-L/14 and passed through a multi-layer perceptron for classification.

Output: Binary classification prediction (real vs. synthetic) for detection. For flaw classification, 14 independent binary labels corresponding to specific synthetic artifact types.

Scoring recipe

def compute_accuracy(preds, gold):
    return (preds == gold).mean() * 100

def compute_em(preds, gold):
    # preds and gold are (N, 14) binary arrays
    return (preds == gold).all(axis=1).mean() * 100

def compute_mAP(preds, gold):
    # preds and gold are (N, 14) arrays of scores/labels
    aps = [average_precision_score(gold[:, i], preds[:, i]) for i in range(14)]
    return sum(aps) / len(aps)

Common pitfalls

• Table 1 reports accuracy as 'hard/original' (prior/posterior), which reverses the standard convention and is easily misread.
• GENHARD only contains samples that previous methods misclassified, so accuracy can drop below 50% (worse than random guessing).
• Cross-validation is strictly cross-generator (train on one generator, test on another), not cross-dataset or cross-split.

Evidence (verbatim from paper)

We evaluate our results on GenImage [50], a million-scale dataset covering 8 generator subsets: Midjourney [27], Stable Diffusion V1.4 [33], Stable Diffusion V1.5 [33], ADM [7], GLIDE [28], Wukong [45], VQDM [10], and BigGAN [2]. Due to computational resource constraints, we partition the validation subsets of GenImage by a 9:1 ratio for training and evaluation. The prior number for each cell marks the test accuracy on the hard samples, while the posterior marks the test accuracy on the original samples.

Citation

@misc{wu2025explainable,
  title={Explainable Synthetic Image Detection through Diffusion Timestep Ensembling},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2503.06201}
}

• arXiv: 2503.06201