Tian Gong St Click Eval

tian-gong-st-click-eval

An Adversarial Imitation Click Model for Information Retrieval — Dai et al. (2021) (arXiv:2104.06077, 2021)

What this evaluates

Evaluates click models on predicting user click sequences and estimating document relevance from search logs. It also measures how well models recover the underlying distribution of real click data (distributional coverage) and perform when document lists are poorly ranked.

Datasets

• TianGong-ST — total 147155; splits: train (117431), val (13154), test (16570); repo http://www.thuir.cn/tiangong-st/

Metrics

• LL (primary) — range: other
  • Log-likelihood over all queries and positions: LL = (1/MN) * sum_{i,t} [C_{i,t} log P_{i,t} + (1-C_{i,t}) log(1-P_{i,t})]. Higher values indicate better click prediction performance.
• PPL — range: other
  • Perplexity at rank t: PPL@t = 2^(-1/N * sum_{i} [C_{i,t} log P_{i,t} + (1-C_{i,t}) log(1-P_{i,t})]). Total PPL is averaged over all positions. Lower values indicate better performance.
• NDCG@k — range: [0, 1]
  • Mean Normalized Discounted Cumulative Gain at truncation levels k=1,3,5,10, computed using human-annotated relevance labels on the ranked document list.
• Reverse PPL — range: other
  • Perplexity of a surrogate model trained on synthetic click sequences generated by the target model, evaluated on held-out real data. Lower values indicate better distributional coverage.
• Forward PPL — range: other
  • Perplexity of a surrogate model trained on held-out real data, evaluated on synthetic click sequences generated by the target model. Lower values indicate better distributional coverage.

Input / output format

Input: Query, ranked document list, and historical click sequence for each search session.

Output: Predicted click probability for each document in the list, or a binary click sequence sampled from the model's distribution.

Scoring recipe

def compute_metrics(predictions, golds, relevance_labels, k_values=[1,3,5,10]):
    LL = 0.0
    PPL_sum = 0.0
    N, M = len(predictions), len(predictions[0])
    for i in range(N):
        for t in range(M):
            p = predictions[i][t]
            c = golds[i][t]
            LL += c * log(p) + (1-c) * log(1-p)
            PPL_sum += c * log(p) + (1-c) * log(1-p)
    LL /= (N * M)
    PPL = 2 ** (-PPL_sum / N)
    
    ndcg_scores = []
    for k in k_values:
        scores = [model.estimate_relevance(doc) for doc in doc_list]
        ndcg_scores.append(compute_ndcg(relevance_labels, scores, k))
    return LL, PPL, ndcg_scores

def compute_distributional_ppl(target_model, real_data, surrogate_model, num_samples=7):
    synth_data = target_model.generate_sequences(real_data, num_samples)
    surrogate_model.fit(synth_data) # Reverse PPL
    rev_ppl = surrogate_model.perplexity(real_data)
    surrogate_model.fit(real_data) # Forward PPL
    fwd_ppl = surrogate_model.perplexity(synth_data)
    return rev_ppl, fwd_ppl

Common pitfalls

• Traditional PPL measures one-step prediction accuracy, while Reverse/Forward PPL measure generative distributional coverage; they can yield contradictory rankings for models.
• Reverse/Forward PPL require training surrogate models on synthetic data, which introduces computational overhead and potential bias if surrogate capacity or training epochs are mismatched.
• NDCG is computed using human-annotated labels on only 2,000 query-sessions, not the full test set, so results may not generalize to all queries.

Evidence (verbatim from paper)

For click prediction task, we report the log-likelihood (LL) and perplexity (PPL)... For relevance estimation task, we use click models to rank the document list and calculate the mean Normalized Discounted Cumulative Gain (NDCG)... We report NDCG scores at truncation level 1, 3, 5 and 10.

Citation

@misc{dai2021adversarial,
  title={An Adversarial Imitation Click Model for Information Retrieval},
  author={Dai et al. (2021)},
  year={2021},
  note={arXiv:2104.06077}
}

• arXiv: 2104.06077