Refexp Eval

refexp-eval

Generation and Comprehension of Unambiguous Object Descriptions — Mao et al. (2015) (arXiv:1511.02283, 2015)

What this evaluates

Evaluates a model's ability to generate unambiguous, context-aware text descriptions for specific objects in an image, and to comprehend those descriptions by correctly localizing the target object via bounding box prediction.

Datasets

• G-Ref — total 54822; splits: train (44822), val (5000), test (5000); repo https://github.com/mjhuela/Google_Refexp_toolbox
• UNC-Ref — total 50000; splits: train (40000), val (5000), test (5000); repo https://github.com/mjhuela/Google_Refexp_toolbox

Metrics

• precision@1 (primary) — range: [0, 1]
  • For each image, compute IoU between the predicted bounding box and the ground truth box. If IoU > 0.5, score is 1; else 0. Average across all images.
• human_eval_score — range: percent
  • Percentage of automatically generated descriptions judged by AMT workers as better or equal to human-generated captions for the same object.

Input / output format

Input: Image + text description (comprehension task) or Image + target object bounding box (generation task).

Output: Predicted bounding box coordinates (comprehension) or a natural language description string (generation).

Scoring recipe

def compute_precision_at_1(pred_boxes, gt_boxes):
    ious = [iou(p, g) for p, g in zip(pred_boxes, gt_boxes)]
    scores = [1.0 if iou > 0.5 else 0.0 for iou in ious]
    return sum(scores) / len(scores)

Common pitfalls

• Standard captioning metrics (BLEU, CIDEr) are unreliable for this task as they do not measure semantic uniqueness.
• End-to-end scores can be artificially inflated because generation and comprehension models share parameters, allowing the system to 'communicate' in its own idiosyncratic language.
• Human evaluation requires blinding workers to machine vs. human origin to prevent bias.

Evidence (verbatim from paper)

The comprehension task is easy to evaluate: we simply compute the Intersection over Union (IoU) ratio between the true and predicted bounding box. If IoU exceeds 0.5, we call the detection a true positive, otherwise it is a false positive (this is equivalent to computing the precision@1 measure). We then average this score over all images.

Citation

@misc{mao2015generation,
  title={Generation and Comprehension of Unambiguous Object Descriptions},
  author={Mao et al. (2015)},
  year={2015},
  note={arXiv:1511.02283}
}

• arXiv: 1511.02283