Rectom Eval

rectom-eval

RecToM: A Benchmark for Evaluating Machine Theory of Mind in LLM-based Conversational Recommender Systems — Li et al. (2025) (arXiv:2511.22275, 2025)

What this evaluates

Evaluates machine theory of mind in LLM-based conversational recommender systems by testing cognitive inference (fine/coarse intention, belief) and behavioral prediction (prediction, judgement) for both recommender and seeker roles in dialogue scenarios.

Datasets

• RECTOM — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers across all questions. Calculated as (correct predictions / total questions) × 100.

Input / output format

Input: Dialogue context from a conversational recommender system scenario, followed by a question probing cognitive states or behavioral predictions. Prompted with either zero-shot instructions or Chain-of-Thought ('Let's think step by step.').

Output: Multiple choice answer (single or multiple options). For CoT prompts, the final answer is extracted via string matching from a fixed output format.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

Common pitfalls

• CoT prompting requires strict string matching to extract the final answer from the generated reasoning text.
• Questions may have single or multiple correct options; accuracy calculation must correctly handle multi-label answers.
• Generation temperature is fixed at 0.7, which balances creativity and determinism and should be replicated for fair comparison.

Evidence (verbatim from paper)

Following established practices in the theory of mind literature (Sabour et al. 2024; Kim et al. 2023) we evaluate these models with two types of prompting strategies: (1) vanilla zero-shot prompting directly asks LLMs to select the answer (single or multiple options) without providing any explanation. (2) Chain-of-thought (CoT) prompting, adapted from (Kojima et al. 2022; Wei et al. 2022), in which the model is instructed with the prompt "Let's think step by step." to encourage explicit reasoning. The final answer is then extracted via string matching from a fixed output format. the temperature for all model generations is set to 0.7 to balance creativity and determinism. Table 4: Main results of models on RECTOM (accuracy in %).

Citation

@misc{li2025rectom,
  title={RecToM: A Benchmark for Evaluating Machine Theory of Mind in LLM-based Conversational Recommender Systems},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2511.22275}
}

• arXiv: 2511.22275