Visdial Eval

visdial-eval

Visual Dialog — Abhishek Das et al. (arXiv:1611.08669, 2016)

What this evaluates

Evaluates an AI agent's ability to maintain conversational context, resolve co-references, and ground follow-up questions in visual content. The task requires ranking a set of candidate answers based on an image and dialog history.

Datasets

• VisDial v0.9 — total 123000; splits: train (80000), val (3000), test (40000)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of 1/rank for the correct answer across all instances, where rank is the position of the ground-truth answer in the model's ranked list of 10 options.

Input / output format

Input: An image, a sequence of prior dialog turns (question-answer pairs), a current question, and a fixed set of 10 candidate answer options.

Output: A score or probability for each of the 10 candidate answers, used to produce a ranked list.

Scoring recipe

def compute_metrics(predictions, gold_indices):
    ranks = []
    for scores, gold in zip(predictions, gold_indices):
        sorted_indices = np.argsort(-scores)
        rank = np.where(sorted_indices == gold)[0][0] + 1
        ranks.append(rank)
    mrr = np.mean(1.0 / ranks)
    r1 = np.mean([r <= 1 for r in ranks])
    r5 = np.mean([r <= 5 for r in ranks])
    r10 = np.mean([r <= 10 for r in ranks])
    mean_rank = np.mean(ranks)
    return {'MRR': mrr, 'R@1': r1, 'R@5': r5, 'R@10': r10, 'Mean': mean_rank}

Common pitfalls

• Mean Rank is inversely scaled (lower is better), unlike MRR and Recall@k which are higher-is-better.
• The task is closed-set ranking over 10 predefined options, not open-ended generation; models must output scores for all options.
• Ignoring dialog history significantly degrades performance, as shown by the gap between LF-Q and LF-QH/LF-QIH models.

Evidence (verbatim from paper)

Table 1: Performance of methods on VisDial v0.9, measured by mean reciprocal rank (MRR), recall@k and mean rank. Higher is better for MRR and recall@k, while lower is better for mean rank.

Citation

@misc{das2016visualdialog,
  title={Visual Dialog},
  author={Abhishek Das et al.},
  year={2016},
  note={arXiv:1611.08669}
}

• arXiv: 1611.08669