Timetom Eval

timetom-eval

TimeToM: Temporal Space is the Key to Unlocking the Door of Large Language Models' Theory-of-Mind — Guiyang Hou et al. (2024) (arXiv:2407.01455, 2024)

What this evaluates

Evaluates Large Language Models' Theory of Mind (ToM) reasoning capabilities across reading comprehension and interactive dialogue scenarios. It specifically probes the model's ability to track character beliefs over time, assess answerability, and determine information access, with a strong focus on first-order and higher-order (up to third-order) belief reasoning.

Datasets

• ToMI — total ?; splits: test (-1)
• BigToM — total ?; splits: test (-1)
• FanToM — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions) / (total predictions).
• All* score — range: [0, 1]
  • A robustness metric requiring the model to achieve 100% accuracy across all five ToM question types (Belief, Answerability[List], Answerability[Y/N], Infoaccess[List], Infoaccess[Y/N]) within the benchmark.
• All score — range: [0, 1]
  • A robustness metric requiring the model to correctly answer both list-type and yes/no-type questions for the Answerability and Infoaccess categories.

Input / output format

Input: A narrative context (story or multi-turn dialogue) detailing character interactions and events, followed by a Theory of Mind question (multiple-choice, list-type, or yes/no format).

Output: The model's predicted answer string or selected option corresponding to the provided question.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return correct / len(golds)

def compute_all_star_score(predictions, golds, question_types):
    correct_by_type = {t: 0 for t in set(question_types)}
    total_by_type = {t: 0 for t in set(question_types)}
    for p, g, t in zip(predictions, golds, question_types):
        total_by_type[t] += 1
        if p.strip().lower() == g.strip().lower():
            correct_by_type[t] += 1
    return all(correct_by_type[t] == total_by_type[t] for t in question_types)

Common pitfalls

• Confusing first-order ToM (tracking what a character knows) with higher-order ToM (tracking what character A thinks character B knows), which requires explicit temporal belief state tracking.
• Failing to distinguish between list-type and yes/no question formats, as the All* and All robustness metrics score these formats separately and require correctness in both.
• Not strictly adhering to the specified temperature (0 for GPT-series, 0.3 for Llama2-series) and top_p (0.95) settings, which significantly impacts reproducibility and reported accuracy gains.

Evidence (verbatim from paper)

We evaluate TimeToM within reading comprehension and interactive dialogue scenarios, using ToMI, BigToM, and FanToM benchmarks. ... We use All* and All score from Table [2] to evaluate the ToM reasoning robustness of baselines and TimeToM. We achieve +33.0% ($\times$ 4.8) and +31.3% ($\times$ 4.0) absolute accuracy improvement over the 0-shot and 0-shot-CoT GPT-4 baselines for All* score, which requires correct answers to all five types of ToM questions (Belief, Answerability[List, Y/N], and Infoaccess[List, Y/N]).

Citation

@misc{hou2024timetom,
  title={TimeToM: Temporal Space is the Key to Unlocking the Door of Large Language Models' Theory-of-Mind},
  author={Guiyang Hou et al. (2024)},
  year={2024},
  note={arXiv:2407.01455}
}

• arXiv: 2407.01455