Openfake Eval

openfake-eval

OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection — Livernoche et al. (2025) (arXiv:2509.09495, 2025)

What this evaluates

Binary classification capability for detecting AI-generated images versus real photographs. It probes a model's ability to generalize across diverse generative models (diffusion, transformer-based) and real-world social media distributions.

Datasets

• OpenFake — total 3963000; splits: test (-1); repo https://github.com/vicliv/OpenFake

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• TNR — range: [0, 1]
  • True Negative Rate (Specificity): proportion of actual real images correctly identified as real.
• TPR — range: [0, 1]
  • True Positive Rate (Recall/Sensitivity): proportion of actual fake images correctly identified as fake.

Input / output format

Input: Single image (real photograph or AI-generated/deepfake image).

Output: Binary classification label (real vs. fake) or probability score.

Scoring recipe

tp = sum(p == 1 and g == 1 for p, g in zip(preds, golds))
fp = sum(p == 1 and g == 0 for p, g in zip(preds, golds))
fn = sum(p == 0 and g == 1 for p, g in zip(preds, golds))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• High true positive rates on fakes often mask unacceptably high false positive rates on real images (low TNR), as seen with legacy baselines.
• Models trained on narrow or outdated datasets (e.g., FaceForensics++) fail to generalize to modern diffusion-based generators.
• The in-the-wild social media test set is small (1,220 images) and heavily curated, potentially underestimating real-world difficulty.

Evidence (verbatim from paper)

Table 4: Generalization of SwinV2 detectors trained on different benchmarks when evaluated on an in-the-wild social-media set (1,057 real, 163 fake; see [Section 3]). Metrics include TNR (real) and TPR (fake). Training on OpenFake yields balanced performance, while GenImage and Semi-Truths show strong class biases.

Citation

@misc{livernoche2025openfake,
  title={OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection},
  author={Livernoche et al. (2025)},
  year={2025},
  note={arXiv:2509.09495}
}

• arXiv: 2509.09495