Idsr Eval

idsr-eval

Improving End-to-End Sequential Recommendations with Intent-aware Diversification — Chen et al. (2019) (arXiv:1908.10171, 2019)

What this evaluates

Evaluates the accuracy and diversity of end-to-end sequential recommendation models by testing their ability to predict the next item in a user's behavior sequence while maintaining item diversity in the recommendation list.

Datasets

• ML100K — total 100000; splits: train (-1), val (-1), test (-1)
• ML1M — total 1000209; splits: train (-1), val (-1), test (-1)

Metrics

• Recall (primary) — range: [0, 1]
  • Fraction of ground-truth test items that appear in the top-K recommended list.
• MRR — range: [0, 1]
  • Reciprocal of the rank of the first relevant item in the recommended list.
• ILD — range: other
  • Average pairwise dissimilarity between items in the recommendation list, calculated as the Euclidean distance between their genre vectors.

Input / output format

Input: A sequence of the user's 9 most recent movie interactions, sorted by timestamp.

Output: A ranked list of recommended movies (top-K).

Scoring recipe

def compute_metrics(recommended_list, test_item, genre_vectors):
    recall = 1.0 if test_item in recommended_list else 0.0
    mrr = 0.0
    for rank, item in enumerate(recommended_list, 1):
        if item == test_item:
            mrr = 1.0 / rank
            break
    n = len(recommended_list)
    ild = 0.0
    if n > 1:
        dists = [euclidean_dist(genre_vectors[i], genre_vectors[j]) for i in range(n) for j in range(i+1, n)]
        ild = (2.0 / (n * (n - 1))) * sum(dists)
    return recall, mrr, ild

Common pitfalls

• Diversity evaluation requires item attributes (e.g., genres); datasets containing only IDs cannot be used for diversity metrics.
• The MMR baseline uses a fixed trade-off parameter θ=0.5, which may not be optimal across different datasets.
• Input sequences are strictly truncated to the 9 most recent interactions, potentially ignoring long-term user preferences.

Evidence (verbatim from paper)

For accuracy evaluation, we use Recall and MRR (Li et al. 2017; Liu et al. 2018); For diversity evaluation, we use Intra-List Distance (ILD) (Zhang and Hurley 2008): ... ILD is defined as: $$ \mathrm {I L D} = \frac {2}{| R _ {L} | (| R _ {L} | - 1)} \sum_ {(i, j) \in R _ {L}} d _ {i j}. \tag {12} $$ We calculate the dissimilarity $d_{ij}$ between two movies based on Euclidean distance between the item genre vectors of movies (Ashkan et al. 2015).

Citation

@misc{chen2019improving,
  title={Improving End-to-End Sequential Recommendations with Intent-aware Diversification},
  author={Chen et al. (2019)},
  year={2019},
  note={arXiv:1908.10171}
}

• arXiv: 1908.10171