Clevr Ref Plus Eval

clevr-ref-plus-eval

CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions — Liu et al. (2019) (arXiv:1901.00850, 2019)

What this evaluates

This benchmark evaluates a model's ability to comprehend referring expressions in synthetic visual scenes. It probes compositional visual reasoning by measuring how well models localize objects based on text descriptions that vary in attribute complexity, spatial relationships, and reasoning topology.

Datasets

• CLEVR-Ref+ — total ?; splits: validation (-1)

Metrics

• accuracy — range: [0, 1]
  • Binary metric indicating whether the predicted bounding box exactly matches the ground truth candidate box among given options.
• IoU (primary) — range: [0, 1]
  • Intersection over Union: the area of overlap between the predicted segmentation mask and the ground truth mask divided by the area of their union. Values range from 0 (no overlap) to 1 (perfect overlap).

Input / output format

Input: RGB image (resized to 320x320) and a natural language referring expression.

Output: Bounding box coordinates (for detection models) or a binary segmentation mask (for segmentation models).

Scoring recipe

def score(predictions, golds):
    ious, accs = [], []
    for pred, gold in zip(predictions, golds):
        if pred['type'] == 'detection':
            accs.append(int(pred['box'] == gold['box']))
        else:
            inter = (pred['mask'] & gold['mask']).sum()
            union = (pred['mask'] | gold['mask']).sum()
            ious.append(inter / union if union > 0 else 0.0)
    return {'accuracy': sum(accs)/len(accs), 'iou': sum(ious)/len(ious)}

Common pitfalls

• Confusing detection accuracy with segmentation IoU, as the paper evaluates two distinct model families with different metrics.
• Assuming intermediate reasoning steps are perfectly accurate; the paper notes the 'Unique' module often fails and degrades intermediate IoU by ~0.66.
• Evaluating only on valid expressions; the protocol explicitly tests robustness to false-premise expressions (e.g., 'The red sphere' when none exist), which require zero-foreground predictions.

Evidence (verbatim from paper)

Detection models are evaluated by accuracy (i.e. whether the prediction selects the correct bounding box among given candidates), where MAttNet performs favorably against SLR. Segmentation models are evaluated by Intersection over Union (IoU), where IEP-Ref performs significantly better than RMI.

Citation

@misc{liu2019clevrrefplus,
  title={CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions},
  author={Liu et al. (2019)},
  year={2019},
  note={arXiv:1901.00850}
}

• arXiv: 1901.00850