Zs Cir Eval

zs-cir-eval

SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval — Wu et al. (2025) (arXiv:2509.26330, 2025)

What this evaluates

Evaluates a model's ability to retrieve target images based on a reference image and a natural language modification text. It probes fine-grained visual-semantic alignment, compositional reasoning, and ranking precision under varying levels of distractors and semantic transformations.

Datasets

• CIRR — total ?; splits: test (-1)
• CIRCO — total ?; splits: test (-1)
• FashionIQ — total ?; splits: val (-1)
• GeneCIS — total 8032; splits: test (-1)

Metrics

• Recall@K (primary) — range: percent
  • For each query, checks if the ground-truth image appears in the top-K retrieved results. The metric is the percentage of queries satisfying this condition across the dataset.
• mAP@K — range: percent
  • Mean Average Precision at K. Computes the average precision for each query considering multiple ground-truth images, then averages across all queries up to rank K.
• Recall_Subset@K — range: percent
  • Recall@K computed on a restricted candidate pool of semantically similar images and challenging negatives, rather than the full retrieval set.
• R@10 / R@50 — range: percent
  • Recall at K=10 and K=50, computed separately for each clothing category (Shirt, Dress, Toptee) and averaged across categories.

Input / output format

Input: A reference image, a natural language modification text, and a retrieval candidate pool (images).

Output: A ranked list of candidate images (or top-16 subset for the reranking stage).

Scoring recipe

def compute_recall_at_k(retrieved_indices, gt_idx, k):
    return int(gt_idx in retrieved_indices[:k])

def compute_mAP_at_k(retrieved_indices, gt_indices, k):
    relevant = 0
    sum_prec = 0.0
    for i, idx in enumerate(retrieved_indices[:k]):
        if idx in gt_indices:
            relevant += 1
            sum_prec += relevant / (i + 1)
    return sum_prec / len(gt_indices) if gt_indices else 0.0

# Aggregate over dataset
total_recall = sum(compute_recall_at_k(preds, gt, k) for preds, gt in dataset) / len(dataset)
total_map = sum(compute_mAP_at_k(preds, gts, k) for preds, gts in dataset) / len(dataset)

Common pitfalls

• CIRR and CIRCO results must be submitted to official external evaluation servers; local computation may yield inconsistent results due to hidden test splits or specific ranking protocols.
• FashionIQ results are reported on the validation set, not the test set, which is non-standard for many benchmarks.
• GeneCIS evaluation varies by sub-task: 'Focus on an Attribute' uses a retrieval set of 10 images, while other tasks use 15, affecting absolute Recall@K values.

Evidence (verbatim from paper)

We report Recall@K (R@K) with K = 1, 5, and 10 for CIRR, which evaluates whether the ground-truth image appears among the top-K retrieved results. Additionally, we report Recall${}_{\text{Subset}}$@K with K = 1, 2, and 3 in the subset setting of CIRR... For CIRCO, we adopt the mean Average Precision at K (mAP@K) with K = 5, 10, 25, and 50, which accounts for multiple ground-truths per query... For FashionIQ, we compute R@10 and R@50 separately for each clothing category and report the average across categories. In the case of GeneCIS, we evaluate R@K (K = 1, 2, 3) on four distinct tasks...

Citation

@misc{wu2025square,
  title={SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2509.26330}
}

• arXiv: 2509.26330