qhjqhj00 d-rep-eval
Image Copy Detection for Diffusion Models — Wang et al. (2024) (arXiv:2409.19952, 2024)
What this evaluates
Evaluates a model's ability to detect and quantify the degree of replication between an original image and a diffusion-generated replica. It probes continuous replication level prediction rather than binary copy detection, measuring how well predicted scores align with manually annotated replication levels.
Datasets
- D-Rep — total 40000; splits: test (-1)
Metrics
PCC(primary) — range: percent- Pearson Correlation Coefficient between predicted replication scores and ground truth levels. Computed on cosine similarities scaled by a granularity factor N.
RD— range: percent- Replication Deviation, defined as the mean absolute deviation between predicted and ground truth scores. Also computed on cosine similarities scaled by a granularity factor N.
Input / output format
Input: Image pairs (original, replica) or extracted image features. Models compute cosine similarity between pair features.
Output: Predicted replication score/level (continuous or discrete) for each image pair.
Scoring recipe
def compute_metrics(predictions, ground_truth, N):
scaled_preds = predictions * N
scaled_gt = ground_truth * N
pcc = pearsonr(scaled_preds, scaled_gt)
rd = mean(abs(scaled_preds - scaled_gt))
return pcc, rd
Common pitfalls
- PCC and RD are computed on cosine similarities scaled by a granularity factor N, not raw feature distances.
- RD measures continuous deviation from ground truth levels rather than binary classification accuracy.
- Generalization tests on external diffusion models use only 100 manually labeled pairs per model, not the full D-Rep benchmark.
Evidence (verbatim from paper)
We employ these models as feature extractors and calculate the cosine similarity between pairs of image features (except for GPT-4V Turbo [44], see Section D in the Appendix for the implementation of it). For the computation of PCC and RD, we adjust the granularity by scaling the computed cosine similarities by a factor of $N$.
Citation
@misc{wang2024imagecopydetection,
title={Image Copy Detection for Diffusion Models},
author={Wang et al. (2024)},
year={2024},
note={arXiv:2409.19952}
}
- arXiv: 2409.19952