Diamonds Eval

diamonds-eval

$\texttt{DIAMONDs}$: A Dataset for $\mathbb{D}$ynamic $\mathbb{I}$nformation $\mathbb{A}$nd $\mathbb{M}$ental modeling $\mathbb{O}$f $\mathbb{N}$umeric $\mathbb{D}$iscussions — Ghosh et al. (2025) (arXiv:2505.12651, 2025)

What this evaluates

Evaluates Theory of Mind and participant-centric reasoning in multi-party dialogues by testing a model's ability to track dynamic numerical variables, filter distractors, and reason from specific character perspectives (including false beliefs) rather than using omniscient context.

Datasets

• DIAMONDs — total ?; splits: test (-1); repo https://github.com/StonyBrookNLP/diamonds

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Computed separately for omniscient, participant-centric, true belief, false belief, distractor, and unanswerable subsets.

Input / output format

Input: Multi-party conversational context containing dynamic numerical variables, knowledge asymmetries, and potential distractors, followed by a question that is either omniscient or participant-centric.

Output: A direct answer to the question (exact format not specified, evaluated as correct/incorrect).

Scoring recipe

correct = sum(1 for p, g in zip(predictions, golds) if p == g)
accuracy = (correct / len(golds)) * 100
# Computed separately for subsets: omniscient, participant-centric, true belief, false belief, distractor, unanswerable

Common pitfalls

• Models often fail to filter out irrelevant distractor information when answering participant-centric questions.
• Models struggle with false belief scenarios, frequently answering based on ground truth rather than the participant's outdated knowledge.
• Models exhibit a bias to generate an answer even when questions are underspecified and unanswerable.

Evidence (verbatim from paper)

Table 1: Omniscient is the % of correctly answered omniscient questions, Participant Centric is the % of correctly answered participant-centric ones, and Overall is the accuracy across all the question type.

Citation

@misc{ghosh2025diamonds,
  title={$\texttt{DIAMONDs}$: A Dataset for $\mathbb{D}$ynamic $\mathbb{I}$nformation $\mathbb{A}$nd $\mathbb{M}$ental modeling $\mathbb{O}$f $\mathbb{N}$umeric $\mathbb{D}$iscussions},
  author={Ghosh et al. (2025)},
  year={2025},
  note={arXiv:2505.12651}
}

• arXiv: 2505.12651