Tombench Eval

tombench-eval

Theory of Mind in Large Language Models: Assessment and Enhancement — Chen et al. (2025) (arXiv:2505.00026, 2025)

What this evaluates

Evaluates large language models' Theory of Mind capabilities by testing their ability to infer mental states (beliefs, intentions, emotions) across multiple orders of reasoning using story-based narratives. The benchmark probes whether models can accurately track character perspectives and answer questions about what different agents know or believe in complex social scenarios.

Datasets

• TOMBENCH — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of test samples where the model's selected multiple-choice option exactly matches the ground truth answer. Evaluation is based solely on the final answer option to minimize subjective judgment.

Input / output format

Input: A narrative story describing a scenario involving characters' actions and mental states, followed by a multiple-choice question about the characters' beliefs, intentions, or emotions.

Output: A single selected option from the provided multiple-choice answers.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
metric_value = correct / len(predictions)

Common pitfalls

• Evaluation relies solely on exact match of the final option, which may not reflect true comprehension of the reasoning process.
• The dataset is bilingual (Chinese and English), so evaluation must be conducted separately or carefully aligned for each language to avoid cross-lingual contamination.

Evidence (verbatim from paper)

All test samples in TOMBENCH follow a multiple-choice question answering format, which minimizes subjective judgment. However, the multiple-choice question answering format poses a challenge in determining whether LLMs truly comprehend the questions and answer them correctly, as evaluation is based solely on the final answer option.

Citation

@misc{chen2025theoryofmind,
  title={Theory of Mind in Large Language Models: Assessment and Enhancement},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2505.00026}
}

• arXiv: 2505.00026