Cifake Eval

cifake-eval

CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Images — Bird et al. (2023) (arXiv:2303.14126, 2023)

What this evaluates

Binary classification of real versus AI-generated synthetic images. It probes a model's ability to detect subtle background imperfections and artifacts introduced by latent diffusion models rather than semantic object content.

Datasets

• CIFAKE — total 120000; splits: train (100000), test (20000)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified images out of the total test set. Calculated as (True Positives + True Negatives) / Total.
• precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions. Formula: True positives / (True positives + False positives).
• recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positives. Formula: True positives / (True positives + False negatives).
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall. Formula: 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: 32x32 RGB images.

Output: Binary label: 0 for FAKE (synthetic), 1 for REAL (photograph). Model outputs a sigmoid probability rounded to the nearest integer for inference.

Scoring recipe

def compute_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
    total = len(predictions)
    accuracy = (tp + tn) / total
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• The dataset uses synthetic images generated via Stable Diffusion v1.4 resized to 32x32, which may not generalize to higher-resolution or different diffusion models.
• Classification relies heavily on background imperfections rather than object semantics, so models trained on this may fail on images with clean backgrounds or different generation pipelines.

Evidence (verbatim from paper)

For each class 5,000 images are used for training and 1,000 for testing, i.e. a testing dataset of 16.6%. Within this study, all images from the training dataset are used for the training of positive class “REAL”. Therefore, $50,000$ are used for training and $10,000$ for testing. ... $100,000$ images are used for training ($50,000$ real images and $50,000$ synthetic images), and $20,000$ are used for testing ($10,000$ real and $10,000$ synthetic). ... These 36 artificial neural networks are then compared with regard to classification metrics to derive the topology that performs best. These are the Precision... The Recall... Finally, the F-1 score is considered...

Citation

@misc{bird2023cifake,
  title={CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Images},
  author={Bird et al. (2023)},
  year={2023},
  note={arXiv:2303.14126}
}

• arXiv: 2303.14126