Narm Session Rec Eval

narm-session-rec-eval

Neural Attentive Session-based Recommendation — Jing Li et al. (2017) (arXiv:1711.04725, 2017)

What this evaluates

Evaluates session-based recommendation models by predicting the next item a user will click based on their sequential interaction history within a session. It probes the model's ability to capture both sequential behavior and session-level intent/purpose.

Datasets

• YOOCHOOSE 1/64 — total ?; splits: train (369859), test (55898)
• YOOCHOOSE 1/4 — total ?; splits: train (5917746), test (55898)
• DIGINETICA — total ?; splits: train (719470), test (60858)

Metrics

• Recall@20 (primary) — range: percent
  • The proportion of test cases where the ground-truth next item appears in the top-20 recommended items. It does not consider the actual rank as long as the item is within the top-N.
• MRR@20 — range: percent
  • The average of reciprocal ranks of the desired items across all test cases. The reciprocal rank is set to zero if the item's rank exceeds 20.

Input / output format

Input: A sequence of item IDs representing a user's click history within a session, formatted as [x_1, x_2, ..., x_{n-1}].

Output: A ranked list of the top-20 candidate items for the next click.

Scoring recipe

recalls = []
mrrs = []
for pred_list, gold in test_set:
    top20 = pred_list[:20]
    recalls.append(1.0 if gold in top20 else 0.0)
    if gold in top20:
        rank = top20.index(gold) + 1
        mrrs.append(1.0 / rank)
    else:
        mrrs.append(0.0)
recall_20 = sum(recalls) / len(test_set) * 100
mrr_20 = sum(mrrs) / len(test_set) * 100

Common pitfalls

• YOOCHOOSE is split by time into 1/64 and 1/4 training fractions, which reduces item vocabulary overlap with the test set compared to using the full training set.
• DIGINETICA evaluation filters out test clicks where the target item is absent from the training set, while YOOCHOOSE evaluation retains them to match baseline settings, creating inconsistent strictness across datasets.
• MRR@20 truncates the reciprocal rank to 0 for items ranked beyond position 20, potentially underestimating performance for items ranked just above the cutoff.

Evidence (verbatim from paper)

Recall@20: The primary evaluation metric is Recall@20 that is the proportion of cases when the desired item is amongst the top-20 items in all test cases. Recall@N does not consider the actual rank of the item as long as it is amongst the top-N and also usually correlates well with other metrics such as click-through rate (CTR) [21]. MRR@20 (Mean Reciprocal Rank), which is the average of reciprocal ranks of the desire items. The reciprocal rank is set to zero if the rank is larger than 20.

Citation

@misc{li2017narm,
  title={Neural Attentive Session-based Recommendation},
  author={Jing Li et al. (2017)},
  year={2017},
  note={arXiv:1711.04725}
}

• arXiv: 1711.04725