Kstt Eval

kstt-eval

Knowledge-enhanced Session-based Recommendation with Temporal Transformer — Zhang et al. (2021) (arXiv:2112.08745, 2021)

What this evaluates

Evaluates session-based recommendation models on predicting the next item in a user's click sequence by integrating knowledge graph attributes and temporal dynamics between clicks.

Datasets

• Yoochoose — total ?; splits: Yoochoose1/64 (-1), Yoochoose1/4 (-1)
• Diginetica — total ?; splits: test (-1)
• Last-fm — total ?; splits: test (-1)

Metrics

• Recall@20 (primary) — range: [0, 1]
  • Fraction of test sessions where the ground-truth next item appears in the top-20 recommended items.
• MRR@20 — range: [0, 1]
  • Average of the reciprocal ranks of the first correct item within the top-20 recommendations across all test sessions.

Input / output format

Input: Session sequences of item IDs, augmented with item attributes from a knowledge graph and time intervals between consecutive clicks.

Output: Ranked list of top-20 candidate items for the next click.

Scoring recipe

def compute_metrics(predictions, gold):
    recalls, mrrs = [], []
    for pred, g in zip(predictions, gold):
        if g in pred:
            recalls.append(1.0)
            mrrs.append(1.0 / (pred.index(g) + 1))
        else:
            recalls.append(0.0)
            mrrs.append(0.0)
    return sum(recalls) / len(recalls), sum(mrrs) / len(mrrs)

Common pitfalls

• Datasets use fixed temporal cutoffs for train/test splits, so random shuffling will cause data leakage.
• K is fixed at 20 in the reported tables, but the protocol supports arbitrary K.
• Last-fm requires strict filtering: keep top 40,000 artists and sessions with length between 2 and 50.

Evidence (verbatim from paper)

For the evaluation metrics, we use R@K (Recall@K) and MRR@K (Mean Reciprocal Rank@K) to measure the recommendation performance.

Citation

@misc{zhang2021knowledgeenhanced,
  title={Knowledge-enhanced Session-based Recommendation with Temporal Transformer},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2112.08745}
}

• arXiv: 2112.08745