Rlirank Eval

rlirank-eval

RLIRank: Learning to Rank with Reinforcement Learning for Dynamic Search — Zhou et al. (2021) (arXiv:2105.10124, 2021)

What this evaluates

Evaluates a reinforcement learning framework for dynamic search ranking that adapts to evolving user intents over multiple search iterations using sequential feedback. It also benchmarks standard learning-to-rank performance on static datasets.

Datasets

• TREC 2016 Dynamic Domain — total ?; splits: test (-1)
• TREC 2017 Dynamic Domain — total ?; splits: test (-1)
• MQ2007 — total ?; splits: test (-1)
• MQ2008 — total ?; splits: test (-1)

Metrics

• α-NDCG (primary) — range: [0, 1]
  • Alpha-NDCG, a variant of NDCG that accounts for position bias and user stopping behavior in dynamic search.
• nSDCG — range: [0, 1]
  • Normalized Sequential Discounted Cumulative Gain, measuring ranking quality over sequential search iterations.
• NDCG@k — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff k (k=1,2,3,4,5), standard metric for static learning-to-rank.

Input / output format

Input: Query, document features, and sequential history of retrieved documents and user feedback across search iterations.

Output: Ranked list of documents for each query.

Scoring recipe

def score(predictions, gold, metric, k=None):
    if metric == 'NDCG@k':
        return ndcg_at_k(predictions, gold, k)
    elif metric == 'α-NDCG':
        return alpha_ndcg(predictions, gold)
    elif metric == 'nSDCG':
        return nscg(predictions, gold)
    return mean([score(q_preds, q_gold, metric) for q_preds, q_gold in zip(all_preds, all_gold)])

Common pitfalls

• Confusing α-NDCG with standard NDCG; α-NDCG incorporates position bias and stopping probability.
• Failing to report per-iteration results for dynamic search datasets, which are crucial for evaluating adaptation.
• Using different NDCG cutoffs (k=1 to 5) without specifying which one is the primary metric.

Evidence (verbatim from paper)

Table 1: α-NDCG of baselines, and RLIRank on 2016 TREC Dynamic Domain dataset and nSDCG of ictnet-params2-ns, ictne-emulti, galago-baseline, dqn-5-actions, clip-baseline, and RLIRank on 2017 TREC Dynamic Domain dataset. ... Table 3: NDCG of RankSVM, ListNet, AdaRank-NDCG, MDP, and RLIRank on MQ2007 and MQ2008 dataset.

Citation

@misc{zhou2021rlirank,
  title={RLIRank: Learning to Rank with Reinforcement Learning for Dynamic Search},
  author={Zhou et al. (2021)},
  year={2021},
  note={arXiv:2105.10124}
}

• arXiv: 2105.10124