Diffseg30k Eval

diffseg30k-eval

DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection — Ci et al. (2025) (arXiv:2511.19111, 2025)

What this evaluates

Pixel-level localization of diffusion-based AI edits in images, shifting from whole-image classification to semantic segmentation to identify precisely which regions have been altered by generative models.

Datasets

• DiffSeg30k — total 30000; splits: test (-1)

Metrics

• localization accuracy (primary) — range: [0, 1]
  • Pixel-level semantic segmentation overlap between predicted and ground-truth edit masks, typically computed as mean Intersection-over-Union (mIoU) or Dice coefficient across all edited regions.

Input / output format

Input: RGB images (base images, half real from COCO, half AI-generated) with corresponding ground-truth binary masks indicating the exact regions edited by diffusion models.

Output: Pixel-level binary segmentation mask predicting the edited regions in the input image.

Scoring recipe

def compute_localization_accuracy(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0.0
    return iou
# Aggregate over dataset to compute mIoU

Common pitfalls

• Multi-turn editing compounds artifacts, making ground-truth masks harder to align with model predictions.
• Automated pipeline may produce low-quality edits or mask errors, requiring quality filtering (score < 3) that can bias evaluation if not standardized.
• Cross-generator generalization is tested across 8 different diffusion models, but robustness to image distortions remains a known weakness.

Evidence (verbatim from paper)

It shifts AIGC detection from whole-image classification to pixel-level semantic segmentation, revealing that segmentation models outperform traditional forgery classifiers in both localization accuracy and cross-generator generalization, despite challenges in robustness to image distortions and multi-turn editing complexity.

Citation

@misc{ci2025diffseg30k,
  title={DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection},
  author={Ci et al. (2025)},
  year={2025},
  note={arXiv:2511.19111}
}

• arXiv: 2511.19111