Community Forensics Eval

community-forensics-eval

Community Forensics: Using Thousands of Generators to Train Fake Image Detectors — Park et al. (2024) (arXiv:2411.04125, 2024)

What this evaluates

This evaluation protocol probes the ability of fake image detectors to generalize across a wide variety of generative models and architectures. It measures how well classifiers trained on diverse synthetic data can distinguish real from generated images in both in-distribution and out-of-distribution settings.

Datasets

• Wang et al. [129] — total ?; splits: test (-1)
• Ojha et al. [90] — total ?; splits: test (-1)
• Synthbuster [7] — total ?; splits: test (-1)
• GenImage [137] — total ?; splits: test (-1)
• Community Forensics (Ours) — total ?; splits: test (-1)

Metrics

• mAP (primary) — range: [0, 1]
  • Threshold-independent mean average precision. Computed by averaging the mAP scores across each generative model in the benchmark, rather than globally across all images.
• Acc — range: [0, 1]
  • Standard classification accuracy. Computed by averaging the accuracy scores across each generative model in the benchmark.

Input / output format

Input: Single RGB image resized to 224x224 or 384x384 pixels.

Output: Binary probability score (0 to 1) indicating the likelihood that the image is AI-generated, passed through a sigmoid activation.

Scoring recipe

def compute_metrics(predictions, gold_labels, model_ids):
    model_results = {}
    for model in unique(model_ids):
        mask = [i for i, mid in enumerate(model_ids) if mid == model]
        preds_m = [predictions[i] for i in mask]
        labels_m = [gold_labels[i] for i in mask]
        model_results[model] = {
            'mAP': compute_mAP(preds_m, labels_m),
            'Acc': compute_accuracy(preds_m, labels_m)
        }
    return {
        'mAP': mean([m['mAP'] for m in model_results.values()]),
        'Acc': mean([m['Acc'] for m in model_results.values()])
    }

Common pitfalls

• Averaging metrics per generative model rather than per image, which heavily weights rare models and changes the scale of the final score.
• GenImage's evaluation set is in-distribution for its own classifier but out-of-distribution for others, making direct comparison misleading without context.
• Freezing the pretrained backbone (common in prior work) consistently degrades performance; end-to-end fine-tuning is required for high performance.

Evidence (verbatim from paper)

Following prior works[[90], [129]], we use the threshold-independent mean average precision (mAP) and accuracy (Acc.) as our evaluation metrics. We compute the mAP and accuracy by averaging the results of each generative model. We use five evaluation sets: Wang et al. [129], Ojha et al. [90], Synthbuster[7], GenImage[137], and our evaluation set.

Citation

@misc{park2024communityforensics,
  title={Community Forensics: Using Thousands of Generators to Train Fake Image Detectors},
  author={Park et al. (2024)},
  year={2024},
  note={arXiv:2411.04125}
}

• arXiv: 2411.04125