Photobench Eval

photobench-eval

PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval — Tianyi Xu et al. (2026) (arXiv:2603.01493, 2026)

What this evaluates

Evaluates personalized, intent-driven photo retrieval capabilities that go beyond simple visual matching. It tests a system's ability to fuse multi-source constraints (temporal, spatial, social identity) and correctly abstain when no relevant image exists in a personal album.

Datasets

• PhotoBench — total ?; splits: test (-1); repo https://github.com/LaVieEnRose365/PhotoBench

Metrics

• Recall@K (primary) — range: [0, 1]
  • Fraction of ground-truth relevant images retrieved within the top-K ranked results. Evaluated at K ∈ {1, 5, 10, 20}.
• NDCG@K — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank K, measuring ranked list quality by discounting the relevance of hits at lower positions.
• Precision/Recall/F1 — range: [0, 1]
  • Set-based metrics for variable-length outputs. Precision is the fraction of returned images that are relevant, Recall is the fraction of relevant images retrieved, and F1 is their harmonic mean.
• Reject-Precision/Recall/F1 — range: [0, 1]
  • Metrics for Zero-GT queries measuring abstention ability. Reject-Recall is the proportion of empty-GT queries correctly identified as having no matches, while Reject-Precision measures the reliability of empty responses.

Input / output format

Input: Natural language query/intent (potentially implying spatio-temporal or social constraints) over a personal photo album.

Output: Fixed-length ranked list of up to K images (for embedding models) or a variable-length set of images/empty set (for agents and mobile systems).

Scoring recipe

def recall_at_k(preds, gold, k):
    top_k = set(preds[:k])
    return len(top_k & set(gold)) / len(gold) if gold else 0

def set_metrics(preds, gold):
    tp = len(set(preds) & set(gold))
    prec = tp / len(preds) if preds else 0
    rec = tp / len(gold) if gold else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return prec, rec, f1

def reject_metrics(preds, gold):
    is_empty = len(preds) == 0
    return is_empty, is_empty, is_empty

Common pitfalls

• Ignoring Zero-GT queries: Systems must correctly abstain when no relevant photo exists; returning any image counts as a retrieval hallucination.
• Treating retrieval as purely visual matching: Fails to account for non-visual constraints (timestamps, GPS, social roles) that define intent-driven queries.
• Assuming single-label ground truth: PhotoBench uses one-to-many matches with variable ground truth sizes, requiring set-based or ranking metrics rather than exact-match accuracy.

Evidence (verbatim from paper)

PhotoBench presents two evaluation challenges: (1) it supports one-to-many matches with variable ground truth sizes, and (2) it includes zero-ground-truth (Zero-GT) queries that require system abstention. Hence, we employ two complementary metric families: Top-K Ranking Metrics. Designed for embedding models that output fixed-length lists. We report Recall@K and NDCG@K with K∈{1,5,10,20}, covering the spectrum from best hit to broad shortlists. Set-Based Metrics. Only suitable for hybrid retrieval systems (i.e., Agents and Phones) that return variable-length sets. We evaluate performance across two query types: Normal Query. We report standard Precision, Recall, and F1 to measure the accuracy of the returned image set against the comprehensive ground truth set. Zero-GT Query. To measure systems’ ability to correctly abstain (reject) when no relevant photo exists, we report Reject-Precision, Reject-Recall, and Reject-F1.

Citation

@misc{xu2026photobench,
  title={PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval},
  author={Tianyi Xu et al. (2026)},
  year={2026},
  note={arXiv:2603.01493}
}

• arXiv: 2603.01493