Referring Expression Eval

referring-expression-eval

Modeling Context in Referring Expressions — Yu et al. (2016) (arXiv:1608.00272, 2016)

What this evaluates

This evaluation probes a model's ability to understand visual context by localizing objects described by natural language expressions (comprehension) and generating unambiguous, context-aware descriptions for objects in an image (generation). It specifically tests whether the model leverages intra-category visual comparisons and joint language modeling to produce discriminative referring expressions.

Datasets

• RefCOCO — total ?; splits: testA (-1), testB (-1); repo https://github.com/lichengunc/refer
• RefCOCO+ — total ?; splits: testA (-1), testB (-1); repo https://github.com/lichengunc/refer
• RefCOCOg — total ?; splits: validation (-1); repo https://github.com/lichengunc/refer

Metrics

• IOU (primary) — range: [0, 1]
  • Predict a bounding box for the given expression. Compute Intersection over Union (IOU) ratio between predicted and ground-truth boxes. Count as true positive if IOU > 0.5, else false positive. Average this score over all images.
• BLEU-1 (primary) — range: [0, 1]
  • Standard BLEU metric at n-gram order 1, measuring unigram overlap between generated and reference expressions.
• ROUGE — range: [0, 1]
  • ROUGE-L recall/f1 measuring longest common subsequence overlap between generated and reference expressions.
• METEOR — range: [0, 1]
  • METEOR metric combining unigram precision, recall, and penalty for fragmentation between generated and reference expressions.
• Human Accuracy — range: [0, 1]
  • Three human annotators click the referred object given the image and generated expression. If ≥2 annotators select the true target, the expression is counted as correct. Averaged over images.
• Duplicate Rate — range: [0, 1]
  • Fraction of images for which the model generates the exact same referring expression for multiple objects within that image. Lower is better.

Input / output format

Input: Comprehension: An image and a natural language referring expression. Generation: An image with multiple objects to be described.

Output: Comprehension: A predicted bounding box (coordinates). Generation: A natural language referring expression string.

Scoring recipe

def score_comprehension(pred_boxes, gt_boxes):
    ious = [intersection_over_union(p, g) for p, g in zip(pred_boxes, gt_boxes)]
    return sum(1 for iou in ious if iou > 0.5) / len(ious)

def score_generation(pred_exprs, ref_exprs):
    bleu1 = compute_bleu(1, pred_exprs, ref_exprs)
    rouge = compute_rouge(pred_exprs, ref_exprs)
    meteor = compute_meteor(pred_exprs, ref_exprs)
    return bleu1, rouge, meteor

def score_duplicate_rate(predictions_per_image):
    duplicates = sum(1 for img_preds in predictions_per_image if len(set(img_preds)) < len(img_preds))
    return duplicates / len(predictions_per_image)

Common pitfalls

• Using global image context instead of target-centered or same-category context can degrade comprehension performance, as global context introduces ambiguity.
• Automatic metrics (BLEU/ROUGE) may not correlate well with human judgment for generation quality; human evaluation is required to verify discriminative capability.
• RefCOCOg test set is not publicly released, so evaluation must be restricted to the validation set.

Evidence (verbatim from paper)

In experiments for the referring expression comprehension task, we use the same evaluation as Mao et al[[26]], namely we first predict the region referred by the given expression, then we compute the intersection over union (IOU) ratio between the true and predicted bounding box. If the IOU is larger than 0.5 we count it as a true positive. Otherwise, we count it as a false positive. We average this score over all images. For the referring expression generation task we use automatic evaluation metrics, BLEU, ROUGE, and METEOR developed for evaluating machine translation results, commonly used to evaluate language generation results[[41]][[18]][[5]][[27]][[39]][[23]].

Citation

@misc{yu2016modelingcontext,
  title={Modeling Context in Referring Expressions},
  author={Yu et al. (2016)},
  year={2016},
  note={arXiv:1608.00272}
}

• arXiv: 1608.00272