Referring Expression Generation Eval

referring-expression-generation-eval

Generating Easy-to-Understand Referring Expressions for Target Identifications — Tanaka et al. (2018) (arXiv:1811.12104, 2018)

What this evaluates

This benchmark evaluates a model's ability to generate referring expressions that enable humans to quickly and accurately identify a target object in an image. It prioritizes human comprehension speed and accuracy over purely semantic correctness, particularly for low-salience targets.

Datasets

• RefCOCO — total ?; splits: test_a (-1), test_b (-1)
• RefCOCO+ — total ?; splits: test_a (-1), test_b (-1)
• RefCOCOg — total ?; splits: val (-1)
• RefGTA — total ?; splits: test (-1)

Metrics

• CIDEr — range: [0, 1]
  • Computes the average n-gram similarity between generated sentences and ground-truth references, weighted by TF-IDF scores.
• Meteor — range: [0, 1]
  • Measures translation quality based on unigram precision, recall, and a penalty for word order.
• R1-CIDEr (primary) — range: [0, 1]
  • A ranking-weighted variant of CIDEr. Weights for ground-truth sentences are calculated as $w(r_{ij}) = (rank(r_{ij}) \sum_{j} rank(r_{ij})^{-1})^{-1}$, where rank is derived from human comprehension accuracy and time. The final score is the weighted average similarity.
• Comprehension Accuracy — range: percent
  • The percentage of human annotators who correctly identify the target object given the generated sentence.

Input / output format

Input: An image and a target object (specified by a bounding box or object ID).

Output: A natural language referring expression (sentence) describing the target object.

Scoring recipe

def compute_r1_cider(predictions, golds, human_ranks):
    scores = []
    for pred, gt_list, img_id in zip(predictions, golds, image_ids):
        weights = []
        for j, gt in enumerate(gt_list):
            rank = human_ranks[(img_id, j)]
            denom = sum(1/r for r in [human_ranks[(img_id, k)] for k in range(len(gt_list))])
            w = (rank * denom) ** -1
            weights.append(w)
        score = weighted_cider(pred, gt_list, weights)
        scores.append(score)
    return mean(scores)

Common pitfalls

• Assuming higher CIDEr/Meteor scores always correlate with better human comprehension; the paper explicitly notes that models with higher comprehension accuracy do not always generate sentences with higher automatic metric scores.
• Ignoring the 'impossible to identify' option in human evaluation; annotators were allowed to select this box, and results are reported both including and excluding these cases ('All' vs 'All (selected)').

Evidence (verbatim from paper)

In our study, the ideal metric should assign a high score to a sentence that can be easily comprehended by humans correctly and quickly. While CIDEr calculates the average similarity between a generated sentence from an object $o_{i}$ and ground-truth sentences ${r_{i1},\cdots,r_{im}}$; we define the ranking-weighted CIDEr (R-CIDEr) which utilizes weighted similarity scores between them by the inverse of their rank.

Citation

@misc{tanaka2018referring,
  title={Generating Easy-to-Understand Referring Expressions for Target Identifications},
  author={Tanaka et al. (2018)},
  year={2018},
  note={arXiv:1811.12104}
}

• arXiv: 1811.12104