Ms Cocoai Eval

ms-cocoai-eval

A Comprehensive Dataset for Human vs. AI Generated Image Detection — Roy et al. (2026) (arXiv:2601.00553, 2026)

What this evaluates

Evaluates a model's ability to distinguish real images from AI-generated ones, and to identify the specific generative model that produced a synthetic image. It probes robustness to semantic alignment and fine-grained model attribution.

Datasets

• MS COCOAI — total 96000; splits: test (-1)

Metrics

• baseline_score (primary) — range: [0, 1]
  • Correct predictions divided by total predictions across all test instances.

Input / output format

Input: Image files, optionally paired with their generation captions.

Output: Task A: Binary label ('AI-generated' or 'human-created'). Task B: Categorical label corresponding to one of the five generative models (SD3, SD2.1, SDXL, DALL-E 3, MidJourney v6).

Scoring recipe

def compute_baseline_score(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Semantic alignment via shared captions makes distinguishing real vs. AI images significantly harder than in unaligned datasets.
• Multi-class model attribution (Task B) is substantially more difficult than binary detection, often yielding scores near random chance for similar architectures.
• Performance may drop sharply under systematic image perturbations, requiring robustness testing beyond standard accuracy.

Evidence (verbatim from paper)

Baseline performance metrics, given in Table [3] establish benchmarks for both authenticity detection and model attribution tasks, serving as reference points for subsequent research developments.

Citation

@misc{roy2026cocoai,
  title={A Comprehensive Dataset for Human vs. AI Generated Image Detection},
  author={Roy et al. (2026)},
  year={2026},
  note={arXiv:2601.00553}
}

• arXiv: 2601.00553