Trec Session Tracks Eval

trec-session-tracks-eval

Learning to Personalize for Web Search Sessions — Aloteibi et al. (2020) (arXiv:2009.08206, 2020)

What this evaluates

Evaluates the effectiveness of a learning-to-rank personalization approach for web search sessions by measuring relevance prediction quality across multiple years of session track data. It probes the model's ability to leverage historical query sequences, document rankings, and user click behavior to improve session-level relevance ranking.

Datasets

• TREC 2011-2014 Session Tracks — total 1282; splits: train (-1), val (-1), test (-1)

Metrics

• nDCG@k (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k; computed as DCG@k divided by the ideal DCG@k for the given relevance judgments. DCG sums (2^rel - 1) / log2(rank + 1) over the top-k results.
• nERR@k — range: [0, 1]
  • Normalized Expected Reciprocal Rank at rank k; measures the probability that a user stops at a relevant document within the top-k results, normalized by the ideal session.
• MAP — range: [0, 1]
  • Mean Average Precision; averages the precision values computed at ranks where relevant documents are retrieved across all queries/sessions.

Input / output format

Input: Session context including the sequence of past queries, ranked document lists for past queries, user clicking behavior, and the current test query.

Output: Ranked list of candidate documents (truncated at rank 100) with predicted relevance scores.

Scoring recipe

import math
def compute_ndcg_at_k(pred_scores, rel_labels, k):
    ranked_indices = sorted(range(len(pred_scores)), key=lambda i: pred_scores[i], reverse=True)
    dcg = sum((2**rel_labels[i] - 1) / math.log2(rank + 2) for rank, i in enumerate(ranked_indices[:k]))
    ideal_labels = sorted(rel_labels, reverse=True)
    idcg = sum((2**ideal_labels[i] - 1) / math.log2(i + 2) for i in range(min(k, len(ideal_labels))))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Using standard single-query evaluation instead of session-aware metrics like nERR@k that account for whole-session relevance judgments.
• Failing to use the official TREC evaluation script, which implements specific discounting functions and session-level aggregation conventions.
• Splitting data by documents rather than queries for cross-validation, which causes data leakage in session search benchmarks.

Evidence (verbatim from paper)

The evaluation metrics used in this paper are based on TREC session track's official metrics. These are: nDCG@k, nERR@k and MAP. All runs are evaluated using the official evaluation script.

Citation

@misc{aloteibi2020learning,
  title={Learning to Personalize for Web Search Sessions},
  author={Aloteibi et al. (2020)},
  year={2020},
  note={arXiv:2009.08206}
}

• arXiv: 2009.08206