Casif Eval

casif-eval

Context-aware short-term interest first model for session-based recommendation — Duan et al. (2021) (arXiv:2103.15514, 2021)

What this evaluates

This evaluation protocol assesses a model's ability to perform session-based next-item recommendation by predicting the subsequent item a user will click based on their recent interaction history. It probes the model's capacity to capture both short-term sequential dependencies and long-term contextual patterns within a session without relying on explicit user profiles.

Datasets

• Yoochoose1/64 — total 424757; splits: train (368859), test (55898)
• Yoochoose1/4 — total 5973643; splits: train (5917745), test (55898)
• Diginetica — total 780328; splits: train (719470), test (60858)

Metrics

• Recall@k (primary) — range: percent
  • Recall@k = n_hit / N, where N is the number of test sessions and n_hit is the number of sessions where the ground-truth next item appears in the top-K predicted items.
• MRR@k — range: percent
  • MRR@k = (1/N) * sum(1/rank_i), where rank_i is the position of the ground-truth item in the top-K list (0 if not present).

Input / output format

Input: Chronological sequence of items clicked in a session.

Output: Ranked list of candidate items (top-K) for the next click.

Scoring recipe

def compute_recall_at_k(preds, gold, k):
    hits = sum(1 for p, g in zip(preds, gold) if g in p[:k])
    return hits / len(gold)

def compute_mrr_at_k(preds, gold, k):
    rr = []
    for p, g in zip(preds, gold):
        try:
            rr.append(1.0 / (p[:k].index(g) + 1))
        except ValueError:
            rr.append(0.0)
    return sum(rr) / len(gold)

Common pitfalls

• The standard preprocessing for Yoochoose and Diginetica involves filtering sessions of length 1 and items with fewer than 5 occurrences, plus specific time-based train/test splits; using raw data yields non-comparable results.
• Metrics are reported as percentages in the results tables but defined as proportions in the formulas; failing to multiply by 100 causes a 100x discrepancy.
• Evaluation must be performed at multiple K values (5, 10, 20); reporting only one K value omits critical performance dimensions highlighted in the paper.

Evidence (verbatim from paper)

We use the following performance metrics to compare these algorithms, which have been widely used in session-based recommendation systems.

Recall@k: Be widely used as a measure of predictive accuracy in all kinds of recommendation systems. It represents the proportion of correctly recommended items amongst the top-k items.

$$ \text {R e c a l l} @ k = \frac {n _ {\text {h i t}}}{N}, \tag {14} $$

Where $N$ is the number of test sessions in the testing set, $n_{hit}$ denotes the number of sessions which have hit items among top-K ranking list.

$MRR@k$ : MRR (Mean Reciprocal Rank) is the average of reciprocal ranks of desired items. The reciprocal rank is set to zero if the rank is larger than $K$ .

$$ M R R @ k = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{r a n k _ {i}}, \tag {15} $$

Citation

@misc{duan2021casif,
  title={Context-aware short-term interest first model for session-based recommendation},
  author={Duan et al. (2021)},
  year={2021},
  note={arXiv:2103.15514}
}

• arXiv: 2103.15514