Share Eval

share-eval

Session-based Recommendation with Hypergraph Attention Networks — Wang et al. (2021) (arXiv:2112.14266, 2021)

What this evaluates

Evaluates a model's ability to predict the next item in an anonymous user session based on sequential click history. It probes the model's capacity to capture short-term user intent and higher-order item correlations within dynamic session contexts.

Datasets

• YooChoose — total ?; splits: train (-1), test (-1)
• Diginetica — total ?; splits: train (-1), test (-1)

Metrics

• Hit@20 (primary) — range: percent
  • Binary indicator: 1 if the ground-truth item's rank is ≤ 20, else 0. Averaged over all test sessions.
• MRR@20 — range: percent
  • Reciprocal rank: 1/r_s if ground-truth rank r_s ≤ 20, else 0. Averaged over all test sessions.

Input / output format

Input: A sequence of items clicked by a user within a single anonymous session.

Output: A ranked list of candidate items for the next click, typically truncated to top-K predictions.

Scoring recipe

def evaluate(predictions, ground_truths, K=20):
    hits = 0
    mrrs = 0.0
    for pred_list, gt in zip(predictions, ground_truths):
        rank = pred_list.index(gt) + 1 if gt in pred_list else K + 1
        if rank <= K:
            hits += 1
            mrrs += 1.0 / rank
    return hits / len(predictions) * 100, mrrs / len(predictions) * 100

Common pitfalls

• Leave-one-out evaluation: only the last item in each session is held out as ground truth for testing; all prior items are used as context.
• Evaluation is restricted to the item vocabulary present in the training set; unseen items are excluded from the candidate list.
• K=20 is the standard reporting threshold for overall comparison, while ablation studies also report K=10.
• Statistical significance of improvements is verified using a paired t-test with p < 0.05.

Evidence (verbatim from paper)

As in previous works for session-based recommendation [16, 21, 32], we adopt both Mean Reciprocal Rank (MRR@K) and Hit Rate (Hit@K) as evaluation metrics. Given the ranked list of items predicted for each session, Hit@K measures the probability that the ground-truth item is within the top-K. Let r_s denote the ranking of the ground-truth item for session s. Then, Hit_s@K = 1 if r_s <= K and Hit_s@K = 0 otherwise. As for MRR@K, it measures the average ranking of the ground-truth items among the lists. That is MRR_s@K = 1/r_s if r_s <= K otherwise MRR_s@K = 0. Then we take the average values of MRR and Hit Rate over all the sessions in the test set and report the results.

Citation

@misc{wang2021share,
  title={Session-based Recommendation with Hypergraph Attention Networks},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2112.14266}
}

• arXiv: 2112.14266