Multilingual Tot Sim Eval

multilingual-tot-sim-eval

Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation — He et al. (2026) (arXiv:2604.21096, 2026)

What this evaluates

Evaluates the fidelity of synthetic Tip-of-the-Tongue (ToT) queries by measuring how well they reproduce the relative ranking of retrieval systems compared to real human-authored ToT queries across four languages.

Datasets

• Multilingual ToT Test Collection — total 20000; splits: synthetic (20000), validation_real (600); repo https://github.com/kimdanny/ntcir-19-tot

Metrics

• Kendall's tau & Pearson's r (primary) — range: [-1, 1]
  • Computes the rank correlation between the system rankings induced by real human ToT queries ($Q_{real}$) and synthetic ToT queries ($Q_{syn}$). Kendall's $ au$ measures the proportion of concordant versus discordant pairs, while Pearson's $r$ measures linear correlation between the rank positions.

Input / output format

Input: Wikipedia page text (or LLM-generated summary) and a language-specific prompt template instructing the model to simulate a Tip-of-the-Tongue query for a given entity.

Output: A natural language query in the target language describing the entity, explicitly avoiding the entity's name.

Scoring recipe

# rankings_real: list of system scores/ranks from Q_real
# rankings_syn: list of system scores/ranks from Q_syn
tau = kendalltau(rankings_real, rankings_syn)
r = pearsonr(rankings_real, rankings_syn)
return tau, r

Common pitfalls

• Correlation measures rank order agreement, not absolute metric values (e.g., NDCG scores can differ widely while correlation remains high).
• The validation set for non-English languages includes machine-translated queries, which may introduce noise compared to purely human-authored queries.
• Post-generation anonymity checks discard queries that reveal the entity name, potentially biasing the final set toward entities with more distinctive descriptive features.

Evidence (verbatim from paper)

To quantify the agreement between the two rankings, we compute both Kendall’s $ au$ and Pearson’s $r$ correlation coefficients between $R_{ ext{real}}$ and $R_{ ext{syn}}$. A high correlation indicates that the synthetic queries preserve the relative ordering of retrieval systems observed under real user queries, thereby validating the quality of the simulated ToT queries.

Citation

@misc{he2026multilingual,
  title={Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2604.21096}
}

• arXiv: 2604.21096