Dualrec Movielens Eval

dualrec-movielens-eval

DUALRec: A Hybrid Sequential and Language Model Framework for Context-Aware Movie Recommendation — Li et al. (2025) (arXiv:2507.13957, 2025)

What this evaluates

Evaluates a hybrid sequential and LLM-based framework for next-item movie recommendation. It probes the model's ability to capture temporal user preferences and semantic genre consistency to predict the next movie a user will watch.

Datasets

• MovieLens-1M — total ?; splits: test (-1)

Metrics

• HR@1 — range: [0, 1]
  • Proportion of test cases where the relevant item appears within the top-1 recommendations. Calculated as the number of hits in top-k divided by the total number of test cases.
• HR@5 — range: [0, 1]
  • Proportion of test cases where the relevant item appears within the top-5 recommendations. Calculated as the number of hits in top-k divided by the total number of test cases.
• NDCG@1 — range: [0, 1]
  • Evaluates ranking quality by giving higher weight to correct predictions that appear earlier in the recommendation list, normalized by the ideal DCG.
• NDCG@5 (primary) — range: [0, 1]
  • Evaluates ranking quality by giving higher weight to correct predictions that appear earlier in the recommendation list, normalized by the ideal DCG.
• Genre Jaccard Similarity — range: [0, 1]
  • Computes the Jaccard similarity between the genre tags of the top-1 recommended movie and the user's actual next watched movie to quantify semantic relevance.

Input / output format

Input: User's sequential interaction history and temporal signals, formatted as a natural language prompt for the LLM component, alongside sequential features for the LSTM component.

Output: Top-k ranked list of recommended movies (k=1 for Genre Jaccard, k=5 for HR/NDCG).

Scoring recipe

def compute_metrics(predictions, ground_truth, k=5):
    hits = 0
    ndcg_sum = 0.0
    jaccard_sum = 0.0
    for pred_list, true_item in zip(predictions, ground_truth):
        if true_item in pred_list[:k]:
            hits += 1
        dcg = sum(1.0 / log2(i + 2) for i, item in enumerate(pred_list[:k]) if item == true_item)
        idcg = 1.0 / log2(2)
        ndcg_sum += dcg / idcg
        rec_genres = get_genres(pred_list[0])
        true_genres = get_genres(true_item)
        jaccard_sum += len(rec_genres & true_genres) / len(rec_genres | true_genres)
    return hits / len(predictions), ndcg_sum / len(predictions), jaccard_sum / len(predictions)

Common pitfalls

• Confusing the hard prompt format used in this study with the numerical soft prompts used in the baseline (Xu et al., 2025).
• Assuming Genre Jaccard Similarity is computed over the full top-k list instead of strictly the top-1 recommendation as specified.
• Overlooking that the evaluation constructs natural language prompts from user activity for the LLM, which differs from pure sequential model baselines.

Evidence (verbatim from paper)

To assess the performance of the recommendation model, we employ standard metrics that are often used to assess recommender models (Xu et al., 2025; Wang & Lim, 2023), including Hit Rate at n (HR@(1,5)) and Normalized Discounted Cumulative Gain at n (NDCG@(1,5)). HR@n measures the proportion of test cases where the relevant item appears within the top-n recommendations, calculated as the number of hits in top-k divided by the total number of test cases. NDCG@n evaluates the ranking quality by giving higher weight to correct predictions that appear earlier in the recommendation list.

In addition to ranking accuracy metrics, we also use Genre Jaccard Similarity to assess the semantic understanding of the model’s recommendations. This metric compares the overlap of genre tags between the recommended items and the actual items the user watched next. By doing so, we evaluate how well each DUALRec variant preserves genre-level consistency, providing a complementary perspective on whether the generated movie recommendations align with the thematic preferences reflected in the user’s true viewing behavior. Specifically, we compute the Jaccard similarity between the top-1 recommended movie

Citation

@misc{li2025dualrec,
  title={DUALRec: A Hybrid Sequential and Language Model Framework for Context-Aware Movie Recommendation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2507.13957}
}

• arXiv: 2507.13957