Raid Robustness Eval

raid-robustness-eval

RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors — Eddoubi et al. (2025) (arXiv:2506.03988, 2025)

What this evaluates

Evaluates the adversarial robustness and transferability of AI-generated image detectors against crafted perturbations. It probes whether detectors can maintain classification accuracy when faced with white-box and black-box evasion attacks across different perturbation budgets.

Datasets

• RAID — total 96000; splits: test (96000); repo https://github.com/pralab/RAID

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and true positive rate (TPR). Defined as F1 = 2 * (Precision * TPR) / (Precision + TPR).
• Accuracy — range: [0, 1]
  • Ratio of correctly predicted samples over the total number of samples, using a classification threshold of 0.5.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, plotting True Positive Rate against False Positive Rate across all classification thresholds.

Input / output format

Input: Center-cropped images with detector-specific preprocessing applied. Images are provided as raw tensors or PNG files to avoid lossy compression artifacts.

Output: Binary classification label (real/fake) or continuous probability score. A threshold of 0.5 is applied for accuracy calculation.

Scoring recipe

def compute_metrics(y_true, y_pred_scores, threshold=0.5):
    y_pred = (y_pred_scores >= threshold).astype(int)
    tp = np.sum((y_pred == 1) & (y_true == 1))
    fp = np.sum((y_pred == 1) & (y_true == 0))
    fn = np.sum((y_pred == 0) & (y_true == 1))
    precision = tp / (tp + fp + 1e-8)
    tpr = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * tpr / (precision + tpr + 1e-8)
    accuracy = np.mean(y_pred == y_true)
    auroc = roc_auc_score(y_true, y_pred_scores)
    return {'F1': f1, 'Accuracy': accuracy, 'AUROC': auroc}

Common pitfalls

• Applying lossy compression (e.g., JPEG) to adversarial images degrades perturbation transferability and artificially inflates detector robustness.
• Data drift between the D³ test set and the original training datasets of baseline detectors causes significant performance drops unrelated to adversarial attacks.
• Using a fixed 0.5 threshold for accuracy may mislead evaluation on unbalanced datasets; F1-score or AUROC should be prioritized.

Evidence (verbatim from paper)

To evaluate the performance of the detectors, we make use of the following metrics: F1-score. The F1 score measures the harmonic mean of the precision and true positive rate (TPR), which provides a metric capable of reliably computing the model’s performance in the presence of unbalanced class distributions. It is defined as: F1=2×(Precision×TPR)/(Precision+TPR).

Citation

@misc{eddoubi2025raid,
  title={RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors},
  author={Eddoubi et al. (2025)},
  year={2025},
  note={arXiv:2506.03988}
}

• arXiv: 2506.03988