Durecdial 2.0 Eval

durecdial-2.0-eval

DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation — Liu et al. (2021) (arXiv:2109.08877, 2021)

What this evaluates

Evaluates conversational recommendation systems across monolingual, multilingual, and cross-lingual settings. It probes a model's ability to generate relevant and fluent responses, select correct knowledge entities, maintain topic consistency, and successfully guide dialogues toward a recommendation target.

Datasets

• DuRecDial 2.0 — total 16500; splits: train (-1), dev (-1), test (-1); repo https://github.com/liuzeming01/DuRecDial

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score between the generated response and the ground truth response, measuring overall relevance and overlap.
• BLEU-1 / BLEU-2 — range: [0, 1]
  • Standard BLEU score using unigram (1) and bigram (2) n-gram precisions with a brevity penalty to measure fluency.
• DISTINCT-1 / DISTINCT-2 — range: [0, 1]
  • Ratio of unique n-grams to total n-grams in the generated response, measuring lexical diversity.
• Knowledge Precision / Recall / F1 — range: [0, 1]
  • Precision, recall, and F1 score calculated over the set of knowledge entities selected or used in the generated response compared to the ground truth knowledge set.
• Dialog-Leading Success rate (LS) — range: percent
  • Percentage of dialogues where the model successfully reaches or mentions the recommendation target within a few turns.
• User-Topic Consistency rate (UTC) — range: percent
  • Percentage of dialogues where the model successfully follows new topics introduced by the user.

Input / output format

Input: Concatenated context, goal, and related knowledge as source input, appended with a language identifier token ('EN' or 'ZH') indicating the target response language.

Output: Generated response text in the specified target language.

Scoring recipe

def compute_metrics(predictions, golds, dialogs):
    # Token-level F1
    f1_scores = [token_f1(pred, gold) for pred, gold in zip(predictions, golds)]
    # BLEU & DISTINCT
    bleu1 = nltk.bleu([golds], predictions, weights=(1,0,0,0))
    bleu2 = nltk.bleu([golds], predictions, weights=(0.5,0.5,0,0))
    distinct1 = mean(len(set(pred.split())) / max(len(pred.split()), 1) for pred in predictions)
    distinct2 = mean(len(set(ngrams(pred.split(), 2))) / max(len(pred.split()), 1) for pred in predictions)
    # Knowledge P/R/F1
    gold_ents = [extract_entities(g) for g in golds]
    pred_ents = [extract_entities(p) for p in predictions]
    k_p, k_r, k_f1 = compute_entity_f1(gold_ents, pred_ents)
    # LS & UTC
    ls_rate = sum(1 for d in dialogs if d_reaches_target(d)) / len(dialogs)
    utc_rate = sum(1 for d in dialogs if d_follows_topic(d)) / len(dialogs)
    return {'F1': mean(f1_scores), 'BLEU-1': bleu1, 'BLEU-2': bleu2, 'DIST-1': distinct1, 'DIST-2': distinct2, 'Knowledge P/R/F1': (k_p, k_r, k_f1), 'LS': ls_rate, 'UTC': utc_rate}

Common pitfalls

• News dialogues are explicitly discarded from the train/dev/test splits compared to the original DuRecDial dataset.
• Entity accuracy is critical; incorrect entities in responses heavily penalize Knowledge P/R/F1, LS, UTC, and human evaluation scores, especially for English tasks due to domain mismatch with pretraining corpora.
• Human evaluation uses a 0-2 scale for turn-level and dialog-level metrics (Fluency, Appropriateness, etc.), but Recommendation Success Rate is reported as a percentage.

Evidence (verbatim from paper)

Automatic Evaluation Metrics: For automatic evaluation from the viewpoint of conversation, we follow the setting in previous work (Liu et al., 2020b) to use several common metrics such as F1, BLEU (DLEU1 and DLEU2) (Papineni et al., 2002), and DISTINCT (DIST-1 and DIST-2) (Li et al., 2016) to measure the relevance, fluency, and diversity of generated responses. Moreover, we also evaluate the knowledge-selection capability of each model by calculating knowledge precision/recall/F1 scores as done in Wu et al. (2019); Liu et al., 2020b). In addition, to evaluate rec-ommendation effectiveness, we design two automatic metrics shown as follows. First, to measure how well a model can lead the whole dialog to approach a recommendation target, we design a metric dialog-Leading Success rate (LS). It calculates the percentage of times a dialog can successfully reach or mention the target after a few dialog turns. Second, to measure how well a model can respond to new topics by users, we design a metric User-Topic Consistency rate (UTC). It calculates the percentage of times the model can successfully follow new topics mentioned by users.

Citation

@misc{liu2021durecdial2,
  title={DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation},
  author={Liu et al. (2021)},
  year={2021},
  note={arXiv:2109.08877}
}

• arXiv: 2109.08877