Opensdi Eval

opensdi-eval

OpenSDI: Spotting Diffusion-Generated Images in the Open World — Wang et al. (2025) (arXiv:2503.19653, 2025)

What this evaluates

This benchmark evaluates the ability of vision models to detect and localize diffusion-generated images in an open-world setting. It probes cross-domain generalization across multiple diffusion architectures (SD1.5, SD2.1, SDXL, SD3, Flux.1) and tests robustness against common image degradations like Gaussian blur and JPEG compression.

Datasets

• OpenSDID — total ?; splits: train (-1), test (-1); repo https://github.com/iamwangyabin/OpenSDI

Metrics

• IoU — range: [0, 1]
  • Intersection over Union: ratio of the overlapping area between predicted and ground truth masks to their union area.
• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. Computed separately for pixel-level localization and image-level detection.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified images (real vs. diffusion-generated) out of the total test set.

Input / output format

Input: Single RGB image (resized to 224×224 for feature extraction, 512×512 for augmentation).

Output: Binary classification label (real/fake) for image-level detection, and a binary segmentation mask for pixel-level localization.

Scoring recipe

def compute_metrics(pred_mask, gt_mask, pred_label, gt_label):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0.0
    tp = np.logical_and(pred_mask, gt_mask).sum()
    fp = np.logical_and(pred_mask, np.logical_not(gt_mask)).sum()
    fn = np.logical_and(np.logical_not(pred_mask), gt_mask).sum()
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1_pixel = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    acc = (pred_label == gt_label).mean()
    tp_img = np.logical_and(pred_label, gt_label).sum()
    fp_img = np.logical_and(pred_label, np.logical_not(gt_label)).sum()
    fn_img = np.logical_and(np.logical_not(pred_label), gt_label).sum()
    prec_img = tp_img / (tp_img + fp_img) if (tp_img + fp_img) > 0 else 0.0
    rec_img = tp_img / (tp_img + fn_img) if (tp_img + fn_img) > 0 else 0.0
    f1_img = 2 * prec_img * rec_img / (prec_img + rec_img) if (prec_img + rec_img) > 0 else 0.0
    return {'IoU': iou, 'F1_pixel': f1_pixel, 'F1_img': f1_img, 'Accuracy': acc}

Common pitfalls

• Confusing in-domain (same generator as training) vs. cross-domain (unseen generators like Flux.1, SD3) evaluation splits.
• Mixing pixel-level localization metrics (IoU, F1) with image-level detection metrics (Accuracy, F1) when reporting results.
• Ignoring image degradation robustness tests (Gaussian blur, JPEG compression) which are explicitly evaluated in the paper.

Evidence (verbatim from paper)

Following standard practices in[[12], [37]], we evaluate methods using both pixel-level and image-level metrics on the OpenSDID’s test data (Table [2]). For pixel-level evaluation, we employ the F1-score (F1) and Intersection over Union (IoU). At the image level, we assess performance using F1 and accuracy (Acc).

Citation

@misc{wang2025opensdi,
  title={OpenSDI: Spotting Diffusion-Generated Images in the Open World},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.19653}
}

• arXiv: 2503.19653