Unsupervised Near Duplicate Eval

unsupervised-near-duplicate-eval

Benchmarking unsupervised near-duplicate image detection — Morra et al. (2019) (arXiv:1907.02821, 2019)

What this evaluates

Evaluates the ability of image descriptors to distinguish near-duplicate image pairs from non-duplicates under extreme specificity constraints, simulating large-scale forensic or fraud detection scenarios.

Datasets

• MFND (Mir-Flickr Near-Duplicate) — total ?; splits: test (-1)
• CLAIMS — total ?; splits: test (-1)
• Holidays — total ?; splits: test (-1)
• California-ND — total ?; splits: test (-1)

Metrics

• sensitivity at false positive rate (FPR) (primary) — range: percent
  • Sensitivity = TP / (TP + FN). Evaluated at a fixed, extremely low FPR threshold (e.g., 10^-6 to 10^-9) to ensure high specificity for forensic applications.

Input / output format

Input: Single images processed through pre-trained or fine-tuned CNNs/global descriptors to extract fixed-length feature vectors. Evaluation is performed on image pairs (query, candidate) to compute similarity scores.

Output: Binary prediction (near-duplicate vs. non-duplicate) or similarity score used to rank candidates.

Scoring recipe

def compute_sensitivity_at_fpr(predictions, gold, fpr_threshold):
    tp = sum(1 for p, g in zip(predictions, gold) if p >= threshold and g == 1)
    fn = sum(1 for p, g in zip(predictions, gold) if p < threshold and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p >= threshold and g == 0)
    tn = sum(1 for p, g in zip(predictions, gold) if p < threshold and g == 0)
    sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
    return sensitivity if fpr <= fpr_threshold else None

Common pitfalls

• Hard negative mining can introduce label noise; the paper found ~2-2.4% actual near-duplicates in the negative mining pools that were relabeled.
• Standard accuracy or AUC-ROC can be misleading due to extreme class imbalance and the requirement for ultra-high specificity (10^-9); sensitivity at a fixed low FPR is the correct metric.
• Using different PCA whitening parameters trained on different subsets can cause data leakage if not strictly separated from the test set.

Evidence (verbatim from paper)

reframes the problem as a binary classification task via ROC analysis, showing that fine-tuned CNNs outperform off-the-shelf features, with MFND achieving 96% sensitivity at a false positive rate of 1.43e-6

Citation

@misc{morra2019benchmarking,
  title={Benchmarking unsupervised near-duplicate image detection},
  author={Morra et al. (2019)},
  year={2019},
  note={arXiv:1907.02821}
}

• arXiv: 1907.02821