sid-generalization-eval
Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective — Ouxiang Li et al. (2024) (arXiv:2408.06741, 2024)
What this evaluates
Evaluates a model's ability to generalize synthetic image detection across diverse generative architectures (GANs, Diffusion Models, DiTs) and real-world image sources, focusing on robustness to unseen generators and varying image resolutions.
Datasets
Metrics
ACC (primary) — range: [0, 1]
- Standard classification accuracy: the proportion of correctly classified real versus fake images out of the total test set.
AP — range: [0, 1]
- Average Precision (area under the precision-recall curve) for the binary real/fake classification task.
Input / output format
Input: RGB images, pre-processed via random 256×256 cropping during training and center 256×256 cropping during testing.
Output: Binary classification label: 'real' or 'fake'.
Scoring recipe
def compute_metrics(predictions, labels):
acc = (predictions.argmax(axis=1) == labels).mean()
ap = average_precision_score(labels, predictions)
return acc, ap
Common pitfalls
- Training is strictly limited to ProGAN and LSUN (4-class), so high performance on training data does not indicate generalization.
- The GenImage testset includes highly variable resolutions (128² to 1024²), which can cause performance drops if models are not resolution-invariant.
- Metrics are averaged across 26+ generators (ACC_M / AP_M), masking per-generator weaknesses.
Evidence (verbatim from paper)
The classification accuracy (ACC) and average precision (AP) are introduced as the main metrics in evaluating the SID performance across various generators. To intuitively evaluate the detection performance on GANs and DMs, we also report the averaged metrics for each testset, termed ACC${}{\text{M}}$ and AP${}{\text{M}}$.
Citation
@misc{li2024improving,
title={Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective},
author={Ouxiang Li et al. (2024)},
year={2024},
note={arXiv:2408.06741}
}
1---2name: sid-generalization-eval3description: Evaluates a model's ability to generalize synthetic image detection across diverse generative architectures (GANs, Diffusion Models, DiTs) and real-world image sources, focusing on robustness to unseen generators and varying image resolutions. Use when the user wants to benchmark on SID Generalization Benchmark (ForenSynths, Self-Synthesis, Ojha, GenImage, DiTFake), or asks about evaluating this task. Reports ACC.4---56# sid-generalization-eval78> Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective — Ouxiang Li et al. (2024) (arXiv:2408.06741, 2024)910## What this evaluates1112Evaluates a model's ability to generalize synthetic image detection across diverse generative architectures (GANs, Diffusion Models, DiTs) and real-world image sources, focusing on robustness to unseen generators and varying image resolutions.1314## Datasets1516- **SID Generalization Benchmark (ForenSynths, Self-Synthesis, Ojha, GenImage, DiTFake)** — total ?; splits: train (72000), test (-1); repo https://github.com/Ouxiang-Li/SAFE1718## Metrics1920- `ACC` **(primary)** — range: [0, 1]21 - Standard classification accuracy: the proportion of correctly classified real versus fake images out of the total test set.22- `AP` — range: [0, 1]23 - Average Precision (area under the precision-recall curve) for the binary real/fake classification task.2425## Input / output format2627**Input**: RGB images, pre-processed via random 256×256 cropping during training and center 256×256 cropping during testing.2829**Output**: Binary classification label: 'real' or 'fake'.3031## Scoring recipe3233```python34def compute_metrics(predictions, labels):35 acc = (predictions.argmax(axis=1) == labels).mean()36 ap = average_precision_score(labels, predictions)37 return acc, ap38```3940## Common pitfalls4142- Training is strictly limited to ProGAN and LSUN (4-class), so high performance on training data does not indicate generalization.43- The GenImage testset includes highly variable resolutions (128² to 1024²), which can cause performance drops if models are not resolution-invariant.44- Metrics are averaged across 26+ generators (ACC_M / AP_M), masking per-generator weaknesses.4546## Evidence (verbatim from paper)4748> The classification accuracy (ACC) and average precision (AP) are introduced as the main metrics in evaluating the SID performance across various generators. To intuitively evaluate the detection performance on GANs and DMs, we also report the averaged metrics for each testset, termed ACC${}_{\text{M}}$ and AP${}_{\text{M}}$.4950## Citation5152```bibtex53@misc{li2024improving,54 title={Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective},55 author={Ouxiang Li et al. (2024)},56 year={2024},57 note={arXiv:2408.06741}58}59```6061- arXiv: 2408.06741
qhjqhj00