Forb Eval

forb-eval

FORB: A Flat Object Retrieval Benchmark for Universal Image Embedding — Pengxiang Wu et al. (2023) (arXiv:2309.16249, 2023)

What this evaluates

Evaluates the quality of universal image embeddings for flat object retrieval across diverse 2D domains (e.g., logos, paintings, currency) under varying visual distortions. It probes both candidate rank accuracy and the matching score margin to assess out-of-distribution generalization.

Datasets

• FORB — total ?; splits: test (-1); repo https://github.com/pxiangwu/FORB/

Metrics

• mAP@5 (primary) — range: percent
  • Mean Average Precision at rank 5. Computes the average precision over all queries considering only the top 5 retrieved candidates.
• t-mAP@5 — range: percent
  • FPR-controlled mean Average Precision at rank 5. Measures the matching score margin by evaluating retrieval accuracy while controlling the false positive rate, indicating how well true positives are separated from irrelevant candidates.

Input / output format

Input: Query and database images resized so the longest side is ≤ 480 pixels while maintaining the original aspect ratio. For top-only methods, multi-scale features are extracted using an image pyramid (3 scales for queries, 7 for database) and aggregated via average-pooling followed by L2 normalization.

Output: Ranked list of retrieved database images for each query, evaluated at the top-5 cutoff.

Scoring recipe

def compute_mAP_at_5(retrieved_ids, gt_ids, k=5):
    top_k = retrieved_ids[:k]
    hits = [1 if img in gt_ids else 0 for img in top_k]
    precisions = [sum(hits[:i+1]) / (i+1) for i in range(k)]
    ap = sum(p * h for p, h in zip(precisions, hits)) / max(sum(hits), 1)
    return ap

def compute_t_mAP_at_5(scores, retrieved_ids, gt_ids, fpr_thresh=0.01):
    # Filter candidates by FPR threshold to control matching score margin
    # Then compute mAP@5 on the filtered set
    pass

Common pitfalls

• Training data overlap: Some baseline models (e.g., CLIP, BLIP, DiHT) were trained on datasets like LAION-5B that may contain duplicate images from the FORB test set, potentially inflating reported accuracy.
• Scale invariance: Top-only methods require multi-scale feature extraction and L2 normalization to perform competitively; single-scale evaluation yields significantly lower accuracy.

Evidence (verbatim from paper)

In Table 4 we report image retrieval accuracy for different methods in terms of mAP@5 and t-mAP@5 (see supplementary material for more results). It can be observed that: ... from t-mAP it can be observed that BoW is better at separating true positives from irrelevant candidates, giving a larger matching score margin.

Citation

@misc{wu2023forb,
  title={FORB: A Flat Object Retrieval Benchmark for Universal Image Embedding},
  author={Pengxiang Wu et al. (2023)},
  year={2023},
  note={arXiv:2309.16249}
}

• arXiv: 2309.16249