Disc21 Eval

disc21-eval

The 2021 Image Similarity Dataset and Challenge — Douze et al. (2021) (arXiv:2106.09672, 2021)

What this evaluates

This benchmark evaluates image copy detection systems under realistic, adversarial conditions. It probes a model's ability to match transformed query images against a large reference database while resisting geometric, color, overlay, and deepfake manipulations. The setup emphasizes scalability and robustness in a high-false-positive-rate, needle-in-haystack search regime.

Datasets

• DISC21 — total 2100000; splits: train (1000000), reference (1000000), dev (50000), test (50000); repo https://github.com/facebookresearch/isc2021

Metrics

• micro Average Precision (primary) — range: [0, 1]
  • Area under the precision-recall curve computed jointly across all queries. Formula: μAP = Σ p(i)Δr(i), where p(i) is precision at rank i, Δr(i) is the recall increment, and N is the total number of returned pairs.

Input / output format

Input: A query image (subjected to manual or automatic transformations) and a reference database of 1 million source images.

Output: A ranked list of pairs (query image, candidate source image from reference set) accompanied by a confidence score. Models may omit queries that are not matched.

Scoring recipe

def compute_micro_ap(predictions, gold_positives, total_positives=10000):
    sorted_preds = sorted(predictions, key=lambda x: x[2], reverse=True)
    tp, fp, ap_sum, prev_recall = 0, 0, 0.0, 0.0
    for q, r, _ in sorted_preds:
        if (q, r) in gold_positives:
            tp += 1
        else:
            fp += 1
        precision = tp / (tp + fp)
        recall = tp / total_positives
        ap_sum += precision * (recall - prev_recall)
        prev_recall = recall
    return ap_sum

Common pitfalls

• Confusing micro-AP with macro-AP (mAP); micro-AP computes a single PR curve over all queries, while macro-AP averages AP per query.
• Ignoring distractor queries; false positives from distractor queries directly penalize precision and thus the micro-AP score.
• Assuming all queries must have a match; the protocol explicitly allows models to return empty lists for distractor queries.

Evidence (verbatim from paper)

We use micro Average Precision to measure performance... It is computed as μAP=∑_{i=1}^{N}p(i)Δr(i)∈[0,1] where p(i) is the precision at position i of the sorted list of pairs, Δr(i) is the difference of recall between position i and i-1, and N is the total number of returned pairs for all queries.

Citation

@misc{douze2021disc21,
  title={The 2021 Image Similarity Dataset and Challenge},
  author={Douze et al. (2021)},
  year={2021},
  note={arXiv:2106.09672}
}

• arXiv: 2106.09672