# Mindgames Eval

> Evaluates large language models' ability to perform higher-order epistemic reasoning and multi-agent belief tracking. It probes whether models can correctly update beliefs based on public announcements and answer True/False questions about agents' knowledge states. Use when the user wants to benchmark on MindGames, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/mindgames-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mindgames-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mindgames-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mindgames-eval

---


# mindgames-eval

> MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic — Sileo et al. (2023) (arXiv:2305.03353, 2023)

## What this evaluates

Evaluates large language models' ability to perform higher-order epistemic reasoning and multi-agent belief tracking. It probes whether models can correctly update beliefs based on public announcements and answer True/False questions about agents' knowledge states.

## Datasets

- **MindGames** — total 400; splits: test (400), train (11200), val (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted True/False labels out of the total number of evaluated instances.

## Input / output format

**Input**: A natural language premise describing a multi-agent epistemic scenario, followed by a question and hypothesis formatted as: '<PREMISE> Question: <HYPOTHESIS> True or False ?'

**Output**: The model must generate exactly 'True' or 'False' as the next token/continuation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models may exploit lexical shortcuts or hypothesis-only bias rather than performing genuine belief tracking, as evidenced by the deberta-small baseline achieving 83% accuracy.
- Evaluators must strictly adhere to the lm-eval-harness prompt template to ensure perplexity-based scoring is comparable across models.
- The test set is filtered by difficulty using deberta-small confidence and prediction discrepancy; evaluating on the unfiltered raw generation set may yield inflated performance.

## Evidence (verbatim from paper)

> We present 50 test samples per setup to two NLP researchers only instructed to perform entailment detection. Inter-annotator agreement is 0.89, and average accuracy is 94%.

## Citation

```bibtex
@misc{sileo2023mindgames,
  title={MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic},
  author={Sileo et al. (2023)},
  year={2023},
  note={arXiv:2305.03353}
}
```

- arXiv: 2305.03353