Synthetic Image Detector Eval

synthetic-image-detector-eval

Present and Future Generalization of Synthetic Image Detectors — Bernabeu-Pérez et al. (2024) (arXiv:2409.14128, 2024)

What this evaluates

Evaluates the generalization capability of synthetic image detectors across different generative models, image resolutions, and real-world sources. It probes whether detectors rely on dataset-specific artifacts or scale-dependent biases rather than learning robust forgery signatures.

Datasets

• SuSy Benchmarking Datasets — total ?; splits: test (-1); HF HPAI-BSC/SuSy-Dataset; repo https://github.com/HPAI-BSC/SuSy

Metrics

• recall (primary) — range: [0, 1]
  • True positives divided by the sum of true positives and false negatives. Used when evaluating on a single dataset to focus on identifying the target class without misleading interpretations from class imbalance.
• macro accuracy — range: [0, 1]
  • Unweighted mean of per-class accuracy across all classes in a multi-dataset classification scenario. Ensures fair evaluation regardless of sample size differences between datasets.

Input / output format

Input: Five 224×224 image patches per image, selected based on highest contrast in their grey-level co-occurrence matrix. Patches are processed individually through the network.

Output: Per-patch binary predictions (real vs. synthetic) aggregated to an image-level decision.

Scoring recipe

# Single-dataset evaluation (recall)
tp = sum(pred == 1 and gold == 1)
fn = sum(pred == 0 and gold == 1)
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0

# Multi-dataset evaluation (macro accuracy)
per_class_acc = []
for dataset in datasets:
    correct = sum(pred == gold for pred, gold in zip(dataset.predictions, dataset.ground_truth))
    per_class_acc.append(correct / len(dataset))
macro_accuracy = sum(per_class_acc) / len(per_class_acc)

Common pitfalls

• Image resizing significantly alters detection accuracy depending on the generator, exposing a scaling bias vulnerability that is often overlooked in standard benchmarks.
• Detectors trained on open-source models (e.g., Stable Diffusion) perform poorly on private models (e.g., DALL·E, Midjourney), highlighting a source dependency issue that limits real-world deployment.
• Patch aggregation strategy heavily influences final image-level performance, but the paper notes various combination strategies were studied without a single default.

Evidence (verbatim from paper)

We employ recall when evaluating performance on a single dataset (either authentic or synthetic), focusing exclusively on the model’s effectiveness in identifying the class at hand. This avoids misleading interpretations that could arise from metrics considering both positive and negative classes. For multi-dataset classification scenarios, we utilize macro accuracy, which provides an unweighted mean of per-class accuracy, ensuring fair evaluation across all classes regardless of sample size.

Citation

@misc{bernabeu2024present,
  title={Present and Future Generalization of Synthetic Image Detectors},
  author={Bernabeu-Pérez et al. (2024)},
  year={2024},
  note={arXiv:2409.14128}
}

• arXiv: 2409.14128