Mindgames Eval

mindgames-eval

MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic — Sileo et al. (2023) (arXiv:2305.03353, 2023)

What this evaluates

Evaluates large language models' ability to perform higher-order epistemic reasoning and multi-agent belief tracking. It probes whether models can correctly update beliefs based on public announcements and answer True/False questions about agents' knowledge states.

Datasets

• MindGames — total 400; splits: test (400), train (11200), val (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted True/False labels out of the total number of evaluated instances.

Input / output format

Input: A natural language premise describing a multi-agent epistemic scenario, followed by a question and hypothesis formatted as: ' Question: True or False ?'

Output: The model must generate exactly 'True' or 'False' as the next token/continuation.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models may exploit lexical shortcuts or hypothesis-only bias rather than performing genuine belief tracking, as evidenced by the deberta-small baseline achieving 83% accuracy.
• Evaluators must strictly adhere to the lm-eval-harness prompt template to ensure perplexity-based scoring is comparable across models.
• The test set is filtered by difficulty using deberta-small confidence and prediction discrepancy; evaluating on the unfiltered raw generation set may yield inflated performance.

Evidence (verbatim from paper)

We present 50 test samples per setup to two NLP researchers only instructed to perform entailment detection. Inter-annotator agreement is 0.89, and average accuracy is 94%.

Citation

@misc{sileo2023mindgames,
  title={MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic},
  author={Sileo et al. (2023)},
  year={2023},
  note={arXiv:2305.03353}
}

• arXiv: 2305.03353