Statcan Dialogue Eval

statcan-dialogue-eval

The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents — Lu et al. (2023) (arXiv:2304.01412, 2023)

What this evaluates

Evaluates a model's ability to retrieve relevant statistical data tables from a large corpus based on conversational dialogue history, and its ability to generate appropriate agent responses. It probes intent understanding, table-level grounding, and robustness to temporal distribution shifts.

Datasets

• StatCan Dialogue Dataset — total 19379; splits: train (-1), val (-1), test (-1)

Metrics

• recall@10 (primary) — range: [0, 1]
  • Fraction of test instances where the ground truth table title appears in the top-10 retrieved candidates.
• Title Acc. — range: [0, 1]
  • Exact match accuracy where the generated response exactly matches the ground truth table title.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated and reference responses, scaled to [0, 1].

Input / output format

Input: Dialogue history (user-agent turns) and optional table metadata (title, member items, footnotes, basic info).

Output: For retrieval: a ranked list of candidate table titles. For generation: a single text response string.

Scoring recipe

def calc_recall_at_k(preds, golds, k=10):
    hits = sum(1 for p, g in zip(preds, golds) if g in p[:k])
    return hits / len(golds)

def calc_title_acc(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p.strip() == g.strip())
    return correct / len(golds)

Common pitfalls

• Models exhibit significant performance drops on the test split due to temporal drift (new tables and shifted conversation topics not seen in training).
• Context length limits cause truncation of table metadata, drastically hurting retrieval recall when full tables are used instead of concise metadata.
• French split performance is consistently lower due to smaller dataset size, which may confound cross-lingual comparisons.

Evidence (verbatim from paper)

We observe in Table 9 that DPR outperforms TAPAS and TAPAS-NQ by respectively 23.0% and 15.1% in test recall@10. Moreover, TAPAS-NQ achieves a better performance when it only retrieves the title and member items instead of the full table (p = 0.016), likely due to repetitions and truncation due to context size limits.

Citation

@misc{lu2023statcan,
  title={The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents},
  author={Lu et al. (2023)},
  year={2023},
  note={arXiv:2304.01412}
}

• arXiv: 2304.01412