Vdialogue Eval

vdialogue-eval

VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue — Yunshui Li et al. (2023) (arXiv:2309.07387, 2023)

What this evaluates

Evaluates visually-grounded dialogue systems across five core tasks: multi-modal intent prediction, dialog retrieval (text-to-image and image-to-text), dialog state tracking, and response generation. It provides a unified hierarchical scoring metric to compare cross-task performance and generalization.

Datasets

• VisDial — total ?; splits: test (-1)
• PhotoChat — total ?; splits: test (-1)
• MMDialog — total ?; splits: test (-1)
• Image-Chat — total ?; splits: test (-1)

Metrics

• VDscore (primary) — range: [0, 1]
  • A hierarchical evaluation metric based on the Analytic Hierarchy Process (AHP) that aggregates task-specific performance scores into a single comprehensive score using weighted criteria.
• R@1, R@5, R@10 — range: percent
  • Recall at K, measuring the proportion of instances where the ground-truth item appears within the top K ranked predictions.

Input / output format

Input: Multi-modal dialogue context comprising text history and associated images, with task-specific prompts (e.g., candidate sets for retrieval, intent/state labels for prediction, or generation targets).

Output: Task-dependent: ranked list of images or text, predicted intent/state labels, or generated dialogue responses.

Scoring recipe

def compute_vdscore(task_scores):
    # AHP-based weighted aggregation of task-specific metrics
    return ahp_aggregate(task_scores)

def compute_recall_at_k(predictions, gold, k):
    return 1.0 if gold in predictions[:k] else 0.0

Common pitfalls

• VisDial exhibits a distribution bias towards image content, causing models to ignore dialogue context.
• Annotator bias can create spurious causal links between dialogue context and output responses.
• Models struggle to differentiate correct images from visually similar candidates in retrieval tasks.
• Concatenating long dialogue history with short candidate answers equally degrades text retrieval performance.

Evidence (verbatim from paper)

Specifically, we found that our model also achieved consistent improvement in the comprehensive evaluation of VDscore.

Citation

@misc{li2023vdialogue,
  title={VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue},
  author={Yunshui Li et al. (2023)},
  year={2023},
  note={arXiv:2309.07387}
}

• arXiv: 2309.07387