Co Spy Eval

co-spy-eval

CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI — Cheng et al. (2025) (arXiv:2503.18286, 2025)

What this evaluates

Evaluates the ability of AI-generated image detectors to distinguish between real and synthetic images across diverse generative models, lossy compression formats, and real-world in-the-wild sources. It specifically probes out-of-distribution generalization and robustness to common post-processing transformations like JPEG compression, blurring, and noise.

Datasets

• Co-SpyBench — total ?; splits: test (-1)
• Co-SpyBench/in-the-wild — total ?; splits: test (-1)
• AIGCDetectBenchmark — total ?; splits: test (-1)
• GenImage — total ?; splits: test (-1)

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision (AP) is computed as the area under the precision-recall curve across all classification thresholds. It summarizes detector performance across varying confidence levels.
• Accuracy@0.5 — range: [0, 1]
  • Accuracy is calculated as the fraction of correctly classified samples using a fixed decision threshold of 0.5 on the detector's output probability score.

Input / output format

Input: Single RGB image resized to 224×224 pixels. Inputs are optionally subjected to random JPEG compression (quality 75–95, 50% probability) to simulate real-world upload conditions and prevent format overfitting.

Output: Continuous probability score indicating the likelihood of the image being synthetic. A binary prediction is derived by thresholding the score at 0.5.

Scoring recipe

def compute_metrics(predictions, labels):
    # Accuracy@0.5
    preds_binary = [1 if p >= 0.5 else 0 for p in predictions]
    accuracy = sum(p == g for p, g in zip(preds_binary, labels)) / len(labels)
    
    # Average Precision (AP)
    precisions, recalls, _ = precision_recall_curve(labels, predictions)
    ap = np.trapz(precisions, recalls)
    return ap, accuracy

Common pitfalls

• Skipping the 50% probability JPEG compression (quality 75-95) applied to inputs, which significantly impacts artifact-based detectors and alters reported scores.
• Ignoring the strict 224×224 resizing applied to all images, which can degrade high-frequency artifacts crucial for some baseline methods.
• Evaluating on the same generative model used for training (e.g., SD-v1.4) instead of the intended out-of-distribution unseen models, leading to inflated and non-comparable results.

Evidence (verbatim from paper)

We employ four test sets for a comprehensive assessment of the detection performance: (1) AIGCDetectBenchmark*[[71]]* including synthetic images from 16 different generative models (half GANs and half diffusion models) with equal number of real images from the source training set of each generative model; (2) GenImage*[[73]]* consisting of 8 types of synthetic images with real ones; (3) Co-SpyBench; and (4) Co-SpyBench/in-the-wild, introduced in [Appendix Q]. Following the existing settings*[[59], [43], [5]]*, all images are resized to 224 $ imes$ 224 for both training and test phases, to ensure a fair comparison. In addition, we randomly apply JPEG compression with quality ranging from 75 to 95 with 50% possibility to the inputs. ... We use AP (Average Precision) and Acc. (Accuracy) with threshold 0.5, as two main metrics to evaluate the detection performance.

Citation

@misc{cheng2025cospy,
  title={CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI},
  author={Cheng et al. (2025)},
  year={2025},
  note={arXiv:2503.18286}
}

• arXiv: 2503.18286