# Multilingual Tot Sim Eval

> Evaluates the fidelity of synthetic Tip-of-the-Tongue (ToT) queries by measuring how well they reproduce the relative ranking of retrieval systems compared to real human-authored ToT queries across four languages. Use when the user wants to benchmark on Multilingual ToT Test Collection, or asks about evaluating this task. Reports Kendall's tau & Pearson's r.

- Skill: `qhjqhj00/multilingual-tot-sim-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/multilingual-tot-sim-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/multilingual-tot-sim-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/multilingual-tot-sim-eval

---


# multilingual-tot-sim-eval

> Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation — He et al. (2026) (arXiv:2604.21096, 2026)

## What this evaluates

Evaluates the fidelity of synthetic Tip-of-the-Tongue (ToT) queries by measuring how well they reproduce the relative ranking of retrieval systems compared to real human-authored ToT queries across four languages.

## Datasets

- **Multilingual ToT Test Collection** — total 20000; splits: synthetic (20000), validation_real (600); repo https://github.com/kimdanny/ntcir-19-tot

## Metrics

- `Kendall's tau & Pearson's r` **(primary)** — range: [-1, 1]
  - Computes the rank correlation between the system rankings induced by real human ToT queries ($Q_{real}$) and synthetic ToT queries ($Q_{syn}$). Kendall's $	au$ measures the proportion of concordant versus discordant pairs, while Pearson's $r$ measures linear correlation between the rank positions.

## Input / output format

**Input**: Wikipedia page text (or LLM-generated summary) and a language-specific prompt template instructing the model to simulate a Tip-of-the-Tongue query for a given entity.

**Output**: A natural language query in the target language describing the entity, explicitly avoiding the entity's name.

## Scoring recipe

```python
# rankings_real: list of system scores/ranks from Q_real
# rankings_syn: list of system scores/ranks from Q_syn
tau = kendalltau(rankings_real, rankings_syn)
r = pearsonr(rankings_real, rankings_syn)
return tau, r
```

## Common pitfalls

- Correlation measures rank order agreement, not absolute metric values (e.g., NDCG scores can differ widely while correlation remains high).
- The validation set for non-English languages includes machine-translated queries, which may introduce noise compared to purely human-authored queries.
- Post-generation anonymity checks discard queries that reveal the entity name, potentially biasing the final set toward entities with more distinctive descriptive features.

## Evidence (verbatim from paper)

> To quantify the agreement between the two rankings, we compute both Kendall’s $	au$ and Pearson’s $r$ correlation coefficients between $R_{	ext{real}}$ and $R_{	ext{syn}}$. A high correlation indicates that the synthetic queries preserve the relative ordering of retrieval systems observed under real user queries, thereby validating the quality of the simulated ToT queries.

## Citation

```bibtex
@misc{he2026multilingual,
  title={Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2604.21096}
}
```

- arXiv: 2604.21096