Cops Ref Eval

cops-ref-eval

Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension — Chen et al. (2020) (arXiv:2003.00403, 2020)

What this evaluates

Probes a model's ability to perform compositional visual reasoning and ground referring expressions in the presence of semantically similar distractors. It evaluates whether models can parse complex logical structures and distinguish fine-grained visual differences rather than relying on statistical biases.

Datasets

• Cops-Ref — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Calculated as the percentage of expressions for which the model correctly selects the target region from the candidate set.

Input / output format

Input: A natural language referring expression and a set of candidate images containing the target and semantically similar distractors.

Output: A single selected image/region proposal from the candidate set.

Scoring recipe

correct = 0
for expr, candidates, gt_idx in dataset:
    scores = model.score(expr, candidates)
    pred_idx = argmax(scores)
    if pred_idx == gt_idx:
        correct += 1
accuracy = (correct / len(dataset)) * 100

Common pitfalls

• Models often exploit dataset statistics (e.g., frequent object categories) rather than true compositional reasoning, especially in the 'WithoutDist' setting.
• Performance drops dramatically when distractors share the same category/attributes as the target, revealing over-reliance on object/attribute recognition over relational reasoning.
• Evaluation must be reported across specific distractor configurations (Full, DiffCat, Cat, Cat&attr, Cat&cat) as results are highly sensitive to the type of distractors present.

Evidence (verbatim from paper)

Table 3 reports the accuracy of all the baselines and the proposed MattNet-Mine. We consider it as a correct comprehension if the model successfully chooses the proposal pointed by the expression among all the proposals extracted from the similar image set.

Citation

@misc{chen2020copsref,
  title={Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension},
  author={Chen et al. (2020)},
  year={2020},
  note={arXiv:2003.00403}
}

• arXiv: 2003.00403