Open Sdi Eval

open-sdi-eval

Bridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization — Xiaojie Liang et al. (arXiv:2604.12341, 2026)

What this evaluates

Evaluates a model's ability to localize manipulated regions in images generated by diffusion models, measuring both pixel-level segmentation precision and image-level binary detection capability across multiple unseen generators.

Datasets

• OpenSDI — total ?; splits: train (-1), test (-1)

Metrics

• pixel-level F1 (primary) — range: [0, 1]
  • Standard F1 score computed between predicted and ground-truth binary masks after thresholding predictions at 0.5.
• IoU — range: [0, 1]
  • Intersection over Union between thresholded (at 0.5) predicted and ground-truth masks.
• image-level F1 — range: [0, 1]
  • F1 score for binary image-level detection (fake vs real).
• Accuracy — range: [0, 1]
  • Proportion of correctly classified images (fake vs real).

Input / output format

Input: RGB images resized to 512×512 for the frequency pathway and 224×224 for the semantic pathway.

Output: Per-image binary prediction (real/fake) and a pixel-level probability mask for manipulation localization.

Scoring recipe

def score(pred_masks, gt_masks, pred_images, gt_images):
    pred_masks_bin = (pred_masks > 0.5).astype(int)
    intersection = (pred_masks_bin & gt_masks).sum()
    union = (pred_masks_bin | gt_masks).sum()
    iou = intersection / (union + 1e-8)
    tp, fp, fn = intersection, pred_masks_bin.sum() - intersection, gt_masks.sum() - intersection
    f1 = 2 * tp / (2 * tp + fp + fn + 1e-8)
    pred_label = (pred_images > 0.5).astype(int)
    acc = (pred_label == gt_images).mean()
    tp_i, fp_i, fn_i = (pred_label & gt_images).sum(), (pred_label & (1 - gt_images)).sum(), ((1 - pred_label) & gt_images).sum()
    f1_i = 2 * tp_i / (2 * tp_i + fp_i + fn_i + 1e-8)
    return {'pixel_f1': f1, 'pixel_iou': iou, 'image_f1': f1_i, 'image_acc': acc}

Common pitfalls

• Thresholding predicted localization masks at exactly 0.5 before computing F1 and IoU.
• Evaluating cross-generator generalization without fine-tuning on unseen diffusion models.
• Using different input resolutions (224×224 vs 512×512) for different model branches during evaluation.

Evidence (verbatim from paper)

Pixel-level localization is evaluated with pixel-level F1 and IoU after thresholding the predicted masks at 0.5, while image-level detection is evaluated with image-level F1 and Accuracy.

Citation

@misc{liang2026bridging,
  title={Bridging the Micro--Macro Gap: Frequency-Aware Semantic Alignment for Image Manipulation Localization},
  author={Xiaojie Liang et al.},
  year={2026},
  note={arXiv:2604.12341}
}

• arXiv: 2604.12341