Image Chat Eval

image-chat-eval

Image Chat: Engaging Grounded Conversations — Shuster et al. (2018) (arXiv:1811.00945, 2018)

What this evaluates

Evaluates multimodal conversational models on their ability to generate or retrieve engaging, style-conditioned responses grounded in images and dialogue history. It probes retrieval accuracy, generation quality, and human-perceived engagement in multi-turn image-grounded conversations.

Datasets

• IMAGE-CHAT — total ?; splits: train (-1), test (-1)

Metrics

• R@1 (primary) — range: percent
  • Recall at rank 1: fraction of samples where the true label is ranked first out of a fixed pool of 100 candidates (1 ground truth + 99 random test samples).
• ROUGE-L — range: [0, 1]
  • Standard ROUGE-L recall metric measuring the longest common subsequence between generated and reference utterances.
• BLEU-4 — range: [0, 1]
  • Standard BLEU-4 metric for n-gram precision up to 4-grams.
• Human Win Rate — range: percent
  • Percentage of pairwise comparisons where the model's utterance is preferred over the human's utterance by annotators as 'more engaging'.

Input / output format

Input: Image, style trait, and dialogue history (for turns 2+). For retrieval, a candidate pool of 100 utterances (1 ground truth + 99 random from test set) is provided.

Output: Retrieved utterance from the candidate pool, or a generated text utterance.

Scoring recipe

def calc_r_at_1(predictions, gold):
    # predictions: list of 100 candidate indices ranked by model
    # gold: index of true label in the 100 candidates
    return int(gold in predictions[:1]) / len(predictions)

def calc_rouge_l(generated, reference):
    return compute_rouge_l(generated, reference)

def calc_bleu_4(generated, reference):
    return compute_bleu_4(generated, reference)

def calc_human_win_rate(model_wins, total_comparisons):
    return (model_wins / total_comparisons) * 100

Common pitfalls

• Automatic retrieval evaluation uses a fixed pool of 100 candidates (1 ground truth + 99 random test samples), not the full training set.
• Human evaluations are conducted on a separate set of 500 images not present in the IMAGE-CHAT training data.
• Transfer to the IGC task is performed zero-shot without fine-tuning on the target dataset.

Evidence (verbatim from paper)

We measure recall at 1 and 5 (R@1/100 and R@5/100) retrieval metrics, where for each sample there are 100 candidates to rank: 99 random candidates chosen from the test set, and the true label.

Citation

@misc{shuster2018imagechat,
  title={Image Chat: Engaging Grounded Conversations},
  author={Shuster et al. (2018)},
  year={2018},
  note={arXiv:1811.00945}
}

• arXiv: 1811.00945