Clide Detection Eval

clide-detection-eval

General and Domain-Specific Zero-shot Detection of Generated Images via Conditional Likelihood — Betser et al. (2025) (arXiv:2512.05590, 2025)

What this evaluates

This benchmark evaluates zero-shot detection of AI-generated images across general and domain-specific settings. It probes a model's ability to distinguish real from synthetic images without task-specific fine-tuning, measuring robustness to domain shifts (e.g., artistic styles, damaged cars, invoices) and resistance to 'flipped classification' where detectors misrank generated content as real.

Datasets

• General Image Benchmark (LAION + MS-COCO) — total 200000; splits: test (200000)
• ImaginET — total ?; splits: test (-1)
• CarDD — total 2816; splits: test (2816)
• Invoice Benchmark — total 4000; splits: test (4000)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve; measures the probability that a randomly chosen real image scores higher than a randomly chosen generated image.
• AP — range: [0, 1]
  • Area under the Precision-Recall curve; summarizes classification quality across all thresholds.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall at the calibrated threshold: 2 * (precision * recall) / (precision + recall).
• accuracy — range: [0, 1]
  • Proportion of correctly classified instances (real vs. generated) at the calibrated threshold.

Input / output format

Input: Single RGB image (real or AI-generated).

Output: Continuous criterion value (conditional likelihood score), thresholded to binary prediction (real vs. generated).

Scoring recipe

def evaluate(scores, labels, cal_real_scores):
    th = np.mean(cal_real_scores) + np.std(cal_real_scores)
    preds = (scores >= th).astype(int)
    auc = roc_auc_score(labels, scores)
    ap = average_precision_score(labels, scores)
    f1 = f1_score(labels, preds)
    acc = accuracy_score(labels, preds)
    return {'AUC': auc, 'AP': ap, 'F1': f1, 'Acc': acc}

Common pitfalls

• Flipped classification occurs when AUC < 0.5, meaning the detector consistently assigns higher scores to generated images than real ones, indicating complete failure.
• Threshold calibration must use a held-out set of real images (e.g., 5K MS-COCO validation or 1K domain-specific samples); using test data for thresholding inflates performance.
• Domain shift causes severe baseline degradation; evaluation must be reported per domain (Artistic, Damaged Cars, Invoice) rather than only aggregated.

Evidence (verbatim from paper)

Metrics. We evaluate each detection method using four metrics. To measure the separation between real and generated image distributions, we use two standard metrics: Area Under the Curve (AUC) and Average Precision (AP). Classification performance is further assessed using F1 score and accuracy. The binary classification threshold is set for all methods as $th = \mathrm{mean}(C_{th}) + \mathrm{std}(C_{th})$, where $C_{th}$ are the criterion values of a set of real images.

Citation

@misc{betser2025clide,
  title={General and Domain-Specific Zero-shot Detection of Generated Images via Conditional Likelihood},
  author={Betser et al. (2025)},
  year={2025},
  note={arXiv:2512.05590}
}

• arXiv: 2512.05590