Dddb Eval

dddb-eval

Benchmarking Deepart Detection — Wang et al. (2023) (arXiv:2302.14475, 2023)

What this evaluates

This benchmark evaluates the ability of deep learning models to detect AI-generated art (deeparts) versus conventional art (conarts) and identify their generative origins. It probes detector generalization across different state-of-the-art diffusion models and tests continual learning capabilities under evolving data streams with strict memory constraints.

Datasets

• DDDB — total ?; splits: train (-1), test (-1)

Metrics

• AA (primary) — range: percent
  • Average detection accuracy computed as the mean classification accuracy across all tested generative models or continual learning sessions.
• mAP — range: [0, 1]
  • Mean Average Precision calculated over the binary classification task of distinguishing deeparts from conarts, aggregating precision-recall curves across classes.
• AF — range: percent
  • Average Forgetting degree, measuring the performance drop from the initial training session to the final session in continual learning benchmarks.
• CA — range: [0, 1]
  • Copyright identification accuracy, measuring the model's ability to correctly classify the specific generative model that produced the deepart.

Input / output format

Input: RGB images of either conventional art (conarts) or AI-generated art (deeparts) produced by models such as Stable Diffusion, DALL-E 2, Imagen, Midjourney, or Parti.

Output: Binary classification label (deepart vs. conart) with associated confidence scores, plus class labels for generative origin identification.

Scoring recipe

def compute_metrics(preds, gold, initial_accs, final_accs):
    aa = np.mean(accuracy_score(gold, preds))
    mAP = average_precision_score(gold, preds)
    af = np.mean([init - final for init, final in zip(initial_accs, final_accs)])
    ca = accuracy_score(gold_origin, pred_origin)
    return {'AA': aa, 'mAP': mAP, 'AF': af, 'CA': ca}

Common pitfalls

• Applying traditional deepfake detectors (e.g., trained on ProGAN) directly to deeparts yields near-random accuracy (<50%) due to fundamentally different frequency patterns.
• Standard continual learning methods collapse in zero-buffer settings (CDD3) without the paper's proposed framework (Knowledge Distillation, Cosine Normalization, Prompt-tuning).
• Prompt-tuning (PT) achieves competitive Average Accuracy (AA) but significantly underperforms Fine-tuning (FT) on mAP, requiring careful metric selection for evaluation.

Evidence (verbatim from paper)

using such data augmentation brings marginal improvement in terms of AA, while rising the mAP score by about 3% over the no augmentation case.

Citation

@misc{wang2023benchmarkingdeepart,
  title={Benchmarking Deepart Detection},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2302.14475}
}

• arXiv: 2302.14475