Dvd Dst Eval

dvd-dst-eval

Multimodal Dialogue State Tracking — Le et al. (2022) (arXiv:2206.07898, 2022)

What this evaluates

Evaluates a model's ability to track visual objects and their attributes across turns in video-grounded dialogues. It probes long-term cross-modal dependency resolution and precise state decoding under controlled, bias-free synthetic dialogue conditions.

Datasets

• DVD-DST — total 13992; splits: train (9295), val (3326), test (1371); repo https://github.com/henryhungle/mm.dst

Metrics

• Joint Acc (primary) — range: percent
  • Exact match accuracy over all discrete slot values in a dialogue turn. A prediction is counted as correct only when every component value exactly matches the oracle values.
• Acc IoU @0.5 — range: percent
  • Accuracy for continuous temporal slots (start/end time) using Intersection-over-Union. A prediction is correct if the IoU between the predicted and ground-truth segment exceeds 0.5.
• Obj Ident. F1 — range: percent
  • F1 score computed over object identity predictions across the dataset.

Input / output format

Input: Dialogue context (textual turns) and video context (object-level and segment-level visual features or frames).

Output: A dialogue state represented as a set of tuples (object_id, slot, value) for discrete attributes (size, color, material, shape) and (s_start, s_end) for temporal boundaries.

Scoring recipe

def compute_joint_acc(pred_state, gold_state):
    # pred_state and gold_state are dicts/lists of (slot, value) tuples
    discrete_pred = {k: v for k, v in pred_state if k != 'temporal'}
    discrete_gold = {k: v for k, v in gold_state if k != 'temporal'}
    discrete_match = discrete_pred == discrete_gold
    
    pred_temporal = pred_state.get('temporal', (0, 0))
    gold_temporal = gold_state.get('temporal', (0, 0))
    iou = intersection_over_union(pred_temporal, gold_temporal)
    temporal_match = iou >= 0.5
    
    return 1.0 if discrete_match and temporal_match else 0.0

Common pitfalls

• Existing multimodal dialogue benchmarks suffer from strong distribution and annotator biases, allowing models to ignore dialogue context; DVD-DST was specifically designed to remove these biases.
• Continuous temporal slots require IoU thresholding (0.5 or 0.7) rather than exact match, which can mask poor temporal localization if thresholds are too lenient.
• Component F1 scores can be artificially inflated if models output all possible object identities and slot values, so joint exact-match accuracy is a stricter and more reliable metric.

Evidence (verbatim from paper)

We reported the joint state accuracy of discrete slots only ("Joint Acc") as well as all slot values ("Joint Acc IoU@p"). For continuous slots, we followed (Hu et al., 2016; Gao et al., 2017) by using Intersection-over-Union (IoU) between predicted temporal segment and ground-truth segment. The predicted segment is counted as correct if its IoU with the oracle is more than p, where we chose p = {0.5, 0.7}.

Citation

@misc{le2022multimodal,
  title={Multimodal Dialogue State Tracking},
  author={Le et al. (2022)},
  year={2022},
  note={arXiv:2206.07898}
}

• arXiv: 2206.07898