Long Tail Session Rec Eval

long-tail-session-rec-eval

Long-Tail Session-based Recommendation from Calibration — Chen et al. (2021) (arXiv:2112.02581, 2021)

What this evaluates

This evaluation probes a session-based recommendation model's ability to accurately predict the next item in a user's interaction sequence while mitigating popularity bias. It measures both standard ranking accuracy and the model's capacity to recommend long-tail items, ensuring recommendations align with user-specific item distribution preferences rather than just global popularity.

Datasets

• YOOCHOOSE — total ?; splits: train (-1), val (-1), test (-1)
• Last.fm — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@20 (primary) — range: [0, 1]
  • Computes the proportion of sessions where the ground-truth next item appears in the top-20 recommended items. Formula: (1/|S_te|) * sum(1(x_{n+1} in RL_s)).
• MRR@20 — range: [0, 1]
  • Measures the mean reciprocal rank of the ground-truth item within the top-20 list. If the item is not in the top-20, the score is 0. Formula: (1/|S_te|) * sum(1/rank(x_{n+1}, RL_s)).
• Coverage@20 — range: [0, 1]
  • Represents the proportion of unique items recommended across all test sessions relative to the entire item catalog. Formula: |union(RL_s)| / |I|.
• TailCoverage@20 — range: [0, 1]
  • Similar to Coverage but restricted to long-tail items. Measures the proportion of tail items in the catalog that appear in the top-20 recommendations. Formula: |union(RL_s intersect I_Tail)| / |I_Tail|.
• Tail@20 — range: [0, 1]
  • Calculates the average proportion of tail items in each session's top-20 recommendation list. Formula: (1/|S_te|) * sum(|RL_s intersect I_Tail| / N).
• IP — range: percent
  • Improvement Percentage relative to a baseline model. Formula: (Metric_ourmethod - Metric_othermethod) / Metric_othermethod.

Input / output format

Input: A session sequence of user-item interactions (historical items) used to predict the next item.

Output: A ranked list of top-N (N=20) recommended items.

Scoring recipe

def compute_metrics(predictions, gold_items, tail_items, all_items, N=20):
    recalls, mrrs, cov_items, tail_cov_items, tail_ratios = [], [], set(), set(), []
    for pred_list, gold in zip(predictions, gold_items):
        top_n = pred_list[:N]
        recalls.append(1.0 if gold in top_n else 0.0)
        rank = next((i+1 for i, item in enumerate(top_n) if item == gold), N+1)
        mrrs.append(1.0/rank if rank <= N else 0.0)
        cov_items.update(top_n)
        tail_cov_items.update(top_n & tail_items)
        tail_ratios.append(len(top_n & tail_items) / N)
    return {
        'Recall@20': sum(recalls) / len(recalls),
        'MRR@20': sum(mrrs) / len(mrrs),
        'Coverage@20': len(cov_items) / len(all_items),
        'TailCoverage@20': len(tail_cov_items) / len(tail_items),
        'Tail@20': sum(tail_ratios) / len(tail_ratios)
    }

Common pitfalls

• Tail items are typically defined by purchase frequency (e.g., items with <2 purchases), but the exact threshold is not specified in the text, leading to inconsistent tail definitions across studies.
• The IP (Improvement Percentage) metric is relative to the chosen baseline, so reporting it without explicitly stating the baseline model makes cross-paper comparisons ambiguous.
• Using N=20 for all metrics is fixed here, but many recommendation papers use N=10 or N=50, causing direct numerical comparison issues if not normalized or explicitly noted.

Evidence (verbatim from paper)

Following previous work $ ^{[10, 13]} $ , we use Recall and MRR as evaluation metrics. • Recall@N (Rec@N) is a widely used metric in recommendation and information retrieval areas. Recall@N computes the proportion of correct items in the top-N items of the list. $$ Recall@N=\frac{1}{|S_{te}|}\sum_{s\in S_{te}}1(x_{n+1}\in RL_{s}) $$

Citation

@misc{chen2021longtail,
  title={Long-Tail Session-based Recommendation from Calibration},
  author={Chen et al. (2021)},
  year={2021},
  note={arXiv:2112.02581}
}

• arXiv: 2112.02581