Opentom Eval

opentom-eval

OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models — Hainiu Xu et al. (2024) (arXiv:2402.06044, 2024)

What this evaluates

This benchmark probes Theory-of-Mind (ToM) reasoning in LLMs by testing their ability to infer psychological mental states (e.g., beliefs, attitudes, intentions) and track physical object locations across naturally generated narratives. It specifically evaluates first- and second-order ToM capabilities under varying narrative lengths and question types.

Datasets

• OpenToM — total ?; splits: test (-1)

Metrics

• macro-averaged F1 score (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across binary and ternary classification tasks. It calculates the F1 score for each class independently and then takes the unweighted mean, which is necessary because the ground-truth labels are not uniformly distributed.

Input / output format

Input: A narrative featuring personified characters with explicit intentions and motivated actions, followed by a binary or ternary classification question about physical locations or psychological mental states.

Output: A single classification label (binary or ternary) corresponding to the correct answer for the posed question.

Scoring recipe

def compute_macro_f1(predictions, gold_labels, num_classes):
    f1_scores = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Using accuracy instead of macro-F1 is misleading because the dataset's labels are not uniformly distributed.
• High F1 scores do not guarantee genuine ToM reasoning; the paper notes a high 'unfaithful rate' where models may guess correctly without following the narrative's mental states.
• Applying Chain-of-Thought or SimToM prompting universally can degrade performance on certain question types (e.g., Loc_fine or Att) while helping others.

Evidence (verbatim from paper)

As all the OpenToM questions are formulated as binary or ternary classification tasks and considering that the labels are not uniformly distributed (Figure A3), we evaluate model performance using the macro-averaged F1 scores (referred to as F1 scores henceforth).

Citation

@misc{xu2024opentom,
  title={OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models},
  author={Hainiu Xu et al. (2024)},
  year={2024},
  note={arXiv:2402.06044}
}

• arXiv: 2402.06044