qhjqhj00 sfld-ai-gen-image-detection-eval
SFLD: Reducing the content bias for AI-generated Image Detection — Gye et al. (2025) (arXiv:2502.17105, 2025)
What this evaluates
Evaluates AI-generated image detectors on their ability to generalize across diverse generative models (GANs, diffusion) and resist content bias. It probes robustness using conventional benchmarks, a new content-preserving benchmark (TwinSynths), and low-level vision/perceptual benchmarks to measure how well models rely on texture vs. semantic artifacts.
Datasets
- Conventional benchmark — total ?; splits: test (-1)
- TwinSynths — total ?; splits: test (-1)
- Low-level vision and perceptual benchmarks — total ?; splits: test (-1)
Metrics
AP(primary) — range: percent- Average Precision; a threshold-independent metric computed as the area under the precision-recall curve across all decision thresholds.
Acc.— range: percent- Classification accuracy calculated by applying a fixed decision threshold of 0.5 to model outputs and comparing predicted binary labels with ground truth.
Input / output format
Input: RGB images (real and AI-generated) processed according to each detector's standard input pipeline.
Output: Binary classification prediction (real vs. synthetic) or continuous probability score for each image.
Scoring recipe
def compute_ap(scores, labels):
precisions, recalls, _ = precision_recall_curve(labels, scores)
return auc(recalls, precisions) * 100
def compute_acc(scores, labels):
preds = (scores >= 0.5).astype(int)
return np.mean(preds == labels) * 100
Common pitfalls
- AP is threshold-independent while Acc. uses a fixed 0.5 threshold; comparing them directly without noting the threshold difference is misleading.
- High scores on conventional benchmarks often reflect content bias rather than true generalization, as shown by significant performance drops on TwinSynths.
- All baselines were trained exclusively on a 20-class ProGAN subset, so evaluation tests zero-shot generalization to unseen generators and post-processing methods.
Evidence (verbatim from paper)
We assess the performances of the detection models by average precision score (AP) and classification accuracy (Acc.), following previous works[[49], [32], [46]]. The AP metric is not dependent on the threshold value, whereas the Acc. is calculated with a fixed threshold of 0.5 across all generation models.
Citation
@misc{gye2025sfld,
title={SFLD: Reducing the content bias for AI-generated Image Detection},
author={Gye et al. (2025)},
year={2025},
note={arXiv:2502.17105}
}
- arXiv: 2502.17105