# Opensdi Eval

> This benchmark evaluates the ability of vision models to detect and localize diffusion-generated images in an open-world setting. It probes cross-domain generalization across multiple diffusion architectures (SD1.5, SD2.1, SDXL, SD3, Flux.1) and tests robustness against common image degradations like Gaussian blur and JPEG compression. Use when the user wants to benchmark on OpenSDID, or asks about evaluating this task. Reports F1.

- Skill: `qhjqhj00/opensdi-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/opensdi-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/opensdi-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/opensdi-eval

---


# opensdi-eval

> OpenSDI: Spotting Diffusion-Generated Images in the Open World — Wang et al. (2025) (arXiv:2503.19653, 2025)

## What this evaluates

This benchmark evaluates the ability of vision models to detect and localize diffusion-generated images in an open-world setting. It probes cross-domain generalization across multiple diffusion architectures (SD1.5, SD2.1, SDXL, SD3, Flux.1) and tests robustness against common image degradations like Gaussian blur and JPEG compression.

## Datasets

- **OpenSDID** — total ?; splits: train (-1), test (-1); repo https://github.com/iamwangyabin/OpenSDI

## Metrics

- `IoU` — range: [0, 1]
  - Intersection over Union: ratio of the overlapping area between predicted and ground truth masks to their union area.
- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall. Computed separately for pixel-level localization and image-level detection.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified images (real vs. diffusion-generated) out of the total test set.

## Input / output format

**Input**: Single RGB image (resized to 224×224 for feature extraction, 512×512 for augmentation).

**Output**: Binary classification label (real/fake) for image-level detection, and a binary segmentation mask for pixel-level localization.

## Scoring recipe

```python
def compute_metrics(pred_mask, gt_mask, pred_label, gt_label):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0.0
    tp = np.logical_and(pred_mask, gt_mask).sum()
    fp = np.logical_and(pred_mask, np.logical_not(gt_mask)).sum()
    fn = np.logical_and(np.logical_not(pred_mask), gt_mask).sum()
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1_pixel = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    acc = (pred_label == gt_label).mean()
    tp_img = np.logical_and(pred_label, gt_label).sum()
    fp_img = np.logical_and(pred_label, np.logical_not(gt_label)).sum()
    fn_img = np.logical_and(np.logical_not(pred_label), gt_label).sum()
    prec_img = tp_img / (tp_img + fp_img) if (tp_img + fp_img) > 0 else 0.0
    rec_img = tp_img / (tp_img + fn_img) if (tp_img + fn_img) > 0 else 0.0
    f1_img = 2 * prec_img * rec_img / (prec_img + rec_img) if (prec_img + rec_img) > 0 else 0.0
    return {'IoU': iou, 'F1_pixel': f1_pixel, 'F1_img': f1_img, 'Accuracy': acc}
```

## Common pitfalls

- Confusing in-domain (same generator as training) vs. cross-domain (unseen generators like Flux.1, SD3) evaluation splits.
- Mixing pixel-level localization metrics (IoU, F1) with image-level detection metrics (Accuracy, F1) when reporting results.
- Ignoring image degradation robustness tests (Gaussian blur, JPEG compression) which are explicitly evaluated in the paper.

## Evidence (verbatim from paper)

> Following standard practices in[[12], [37]], we evaluate methods using both pixel-level and image-level metrics on the OpenSDID’s test data (Table [2]). For pixel-level evaluation, we employ the F1-score (F1) and Intersection over Union (IoU). At the image level, we assess performance using F1 and accuracy (Acc).

## Citation

```bibtex
@misc{wang2025opensdi,
  title={OpenSDI: Spotting Diffusion-Generated Images in the Open World},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.19653}
}
```

- arXiv: 2503.19653