Rococo Eval

rococo-eval

RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models — Park et al. (2023) (arXiv:2304.10727, 2023)

What this evaluates

Evaluates the robustness of image-text matching models against adversarial perturbations injected into the retrieval gallery. It probes whether models rely on holistic semantic alignment or are easily misled by locally similar but semantically altered images and captions.

Datasets

• MS-COCO (RoCOCO variant) — total ?; splits: test (-1); repo https://github.com/pseulki/rococo

Metrics

• Recall@1 (primary) — range: percent
  • Percentage of queries for which the ground-truth image or caption is ranked first in the retrieval list after adversarial items are added to the gallery.

Input / output format

Input: Image-text pairs serving as queries, and a gallery set containing original and adversarially perturbed images/captions.

Output: Ranked list of gallery items for each query, sorted by model-computed similarity scores.

Scoring recipe

def compute_recall_at_1(scores, gt_idx):
    ranked = np.argsort(-scores)
    return int(gt_idx in ranked[:1])
# Average over all queries to yield Recall@1 percentage

Common pitfalls

• Adversarial items are injected into the gallery set rather than the query set, fundamentally altering the retrieval task dynamics.
• Source words for caption replacement are selected via low Embedding-Influence scores, meaning perturbations are specifically designed to minimally change the text encoder's output while altering semantics.
• Models often prioritize local textual or visual features over holistic context, leading to severe performance drops on semantically altered but embedding-similar items.

Evidence (verbatim from paper)

revealing that state-of-the-art models like BLIP and VSE∞ suffer severe performance drops (e.g., Recall@1 from 81.9% to 64.5% and 66.1% to 37.5%) despite large-scale pretraining. Our goal is to quantitatively evaluate how well ITM models understand both text and image. Specifically, we measure the robustness of a ITM model through our proposed benchmark, which assesses how robustly the model retrieves the ground-truth image/caption instead of our newly generated adversarial image/caption.

Citation

@misc{park2023rococo,
  title={RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models},
  author={Park et al. (2023)},
  year={2023},
  note={arXiv:2304.10727}
}

• arXiv: 2304.10727