Redial Movie Recommendation Eval

redial-movie-recommendation-eval

Towards Deep Conversational Recommendations — Li et al. (2018) (arXiv:1812.07617, 2018)

What this evaluates

Evaluates modular components of a conversational recommendation system, specifically cold-start movie rating prediction and movie opinion sentiment analysis (seen/liked status) from dialogue text.

Datasets

• REDIAL — total 10000; splits: validation (-1), test (-1)
• MovieLens — total 26000000; splits: train (-1), validation (-1), test (-1)

Metrics

• Cohen's kappa — range: [-1, 1]
  • Measures inter-rater agreement between true labels and predictions, correcting for chance. κ = (p_o - p_e) / (1 - p_e).
• RMSE (primary) — range: [0, 1] or [0, 5]
  • Root Mean Squared Error between predicted and actual ratings. RMSE = sqrt(mean((y_true - y_pred)^2)).

Input / output format

Input: Sentiment: text spans between first and last mention of a movie in a dialogue turn. Recommendation: binary or scaled ratings from a conversation's history (treated as a user profile).

Output: Sentiment: binary classification labels (seen/not seen, liked/not liked). Recommendation: predicted rating value (0/1 or 0.5–5).

Scoring recipe

def compute_kappa(y_true, y_pred):
    return kappa_score(y_true, y_pred)

def compute_rmse(y_true, y_pred):
    return np.sqrt(np.mean((np.array(y_true) - np.array(y_pred)) ** 2))

Common pitfalls

• Class imbalance in sentiment labels (81% liked) requires explicit loss weighting during training.
• Cold-start evaluation treats each conversation as an independent user, ignoring cross-conversation user history.
• MovieLens ratings must be binarized (threshold ≥2) to match REDIAL's distribution before pre-training.

Evidence (verbatim from paper)

We also provide Cohen's kappa coefficient [26] for each model and prediction task. Cohen's kappa measures the agreement between the true label and the predictions. ... Table 2 shows the RMSE obtained on the test set. ... Randomly chosen user-item ratings are held out for validation and test, and only training ratings are used as inputs. Following Sedhain et al. [1], we sampled the training, validation, and test set in a 80-10-10 proportion, and repeated this splitting procedure five times, reporting the average RMSE.

Citation

@misc{li2018towards,
  title={Towards Deep Conversational Recommendations},
  author={Li et al. (2018)},
  year={2018},
  note={arXiv:1812.07617}
}

• arXiv: 1812.07617