Query Suggestion Eval

query-suggestion-eval

Learning to Attend, Copy, and Generate for Session-Based Query Suggestion — Dehghani et al. (2017) (arXiv:1708.03418, 2017)

What this evaluates

Evaluates a session-based query suggestion model's ability to rank candidate queries and generate plausible next queries. It probes the model's discriminative ranking capability and its generative quality in capturing user intent and query reformulation patterns.

Datasets

• Unspecified (replicates Sordoni et al. 2015 setup) — total ?; splits: test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of 1/rank for the correct candidate query across all test instances.
• PER — range: [0, 1]
  • Position Independent Word Error Rate: minimum number of word insertions and deletions to transform the generated query into the target query, normalized by the target query length.
• sim_emb — range: [0, 1]
  • Embedding-based similarity: query-level embeddings computed via vector extrema of pretrained word embeddings, followed by cosine similarity between generated and target query vectors.
• sim_ret+ — range: [0, 1]
  • Retrieval-based similarity using Pseudo Relevance Feedback (PRF) expanded target query as the reference ranked list.
• sim_ret++ — range: [0, 1]
  • Retrieval-based similarity using a merged ranked list from the next half of queries in the session as the reference.

Input / output format

Input: Context session X (sequence of previous queries) and candidate query q for scoring; or context session X for generation.

Output: For discrimination: a scalar score p(q|X). For generation: a generated query string.

Scoring recipe

def compute_metrics(predictions, targets):
    # PER
    per = edit_distance_ignore_order(predictions, targets) / len(targets)
    # sim_emb
    emb_gen = vector_extrema(pretrained_embeddings(predictions))
    emb_tgt = vector_extrema(pretrained_embeddings(targets))
    sim_emb = cosine_similarity(emb_gen, emb_tgt)
    # sim_ret variants
    docs_gen = search_engine.predict(predictions)
    docs_tgt = search_engine.predict(targets)
    sim_ret = ranking_similarity(docs_gen, docs_tgt)
    return per, sim_emb, sim_ret

Common pitfalls

• PER ignores word order, so semantically correct but reordered queries may score poorly.
• Embedding similarity relies on pretrained word vectors and vector extrema, which may miss rare or domain-specific terms.
• Retrieval-based metrics depend on an external document collection and ranking function, making them non-deterministic across different search engines.

Evidence (verbatim from paper)

We use the mean reciprocal rank (MRR) to measure the quality of the ranking. ... As a word overlap based metrics, we consider Position Independent Word Error Rate (denoted as PER), which is the minimum number of word insertions and deletions necessary to transform the generated query into the target query by neglecting the words order, normalized by the length of the target query.

Citation

@misc{dehghani2017learning,
  title={Learning to Attend, Copy, and Generate for Session-Based Query Suggestion},
  author={Dehghani et al. (2017)},
  year={2017},
  note={arXiv:1708.03418}
}

• arXiv: 1708.03418