Omniaid Eval

omniaid-eval

OmniAID: Decoupling Semantic and Artifacts for Universal AI-Generated Image Detection in the Wild — Guo et al. (2025) (arXiv:2511.08423, 2025)

What this evaluates

Evaluates the ability of AI-generated image detectors to generalize across different semantic domains (human, animal, object, scene) and resist modern, photorealistic generative models. It probes whether models rely on content-agnostic artifacts versus semantic features for robust real-vs-fake classification.

Datasets

• GenImage — total ?; splits: test (-1)
• Chameleon — total ?; splits: test (-1)
• Mirage-Test — total ?; splits: test (-1)

Metrics

• Accuracy (%) (primary) — range: percent
  • Standard binary classification accuracy: (True Positives + True Negatives) / Total Samples. Reported per semantic category and as a mean across categories.
• Average Precision (AP) — range: percent
  • Mean Average Precision computed over the precision-recall curve for binary real-vs-fake classification. Reported per category and as a mean.

Input / output format

Input: RGB images resized to 512×512, then downsampled to 336×336 for the CLIP-ViT-L/14 backbone. Each instance is a single image labeled as Real or AI-Generated.

Output: Binary classification label (Real/Fake) or confidence score for each image.

Scoring recipe

def compute_metrics(predictions, labels):
    # predictions: list of predicted labels or probabilities
    # labels: list of ground truth labels (0=Real, 1=Fake)
    accuracy = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
    ap = average_precision_score(labels, predictions)
    return {'accuracy': accuracy, 'ap': ap}

Common pitfalls

• Models often overfit to specific generator fingerprints, causing catastrophic failure on unseen or modern generators (Real/Fake bias).
• Evaluating only on outdated benchmarks like GenImage-SD v1.4 masks poor generalization to contemporary diffusion models.
• The 'Anime' category in Mirage-Test contains only generated samples, making overall accuracy for that category meaningless without separate real/fake reporting.

Evidence (verbatim from paper)

To ensure a fair comparison, we follow the protocol of [genimage, aide], training all models (including our standard OmniAID) exclusively on the GenImage-SD v1.4 dataset to assess generalization from a limited, standard benchmark. Alongside this, to evaluate performance in a realistic, modern scenario, we also train our OmniAID-Mirage model on our modern Mirage-Train dataset. All models are then evaluated on the GenImage [genimage] test set, the in-the-wild Chameleon [aide] dataset, and our new Mirage-Test. Unless otherwise specified, we report classification Accuracy (%) as the primary metric. More Average Precision (AP) results are available in the Supplementary Material.

Citation

@misc{guo2025omniaid,
  title={OmniAID: Decoupling Semantic and Artifacts for Universal AI-Generated Image Detection in the Wild},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2511.08423}
}

• arXiv: 2511.08423