Mus Eval

mus-eval

MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning — Sprague et al. (2023) (arXiv:2310.16049, 2023)

What this evaluates

Evaluates large language models' ability to perform multi-step, commonsense-rich reasoning over long natural language narratives. It probes whether models can follow complex, implicit logical chains (e.g., murder motives, object spatial reasoning, team skill matching) without relying on simple keyword heuristics or rule-based shortcuts.

Datasets

• MuSR — total 756; splits: test (756); repo https://github.com/Zayne-Sprague/MuSR

Metrics

• accuracy (primary) — range: percent
  • Percentage of instances where the model's final predicted answer exactly matches the gold answer. Computed as (number of correct predictions / total instances) * 100.

Input / output format

Input: A natural language narrative (story) containing contextual clues, followed by a question asking for a specific answer (e.g., suspect name, object location, or team assignment). Prompt variants may include chain-of-thought instructions or a textual description of the domain's reasoning strategy (CoT+).

Output: The model must generate a final answer string corresponding to the correct option (e.g., a suspect's name, a room name, or a list of team members). Exact string matching against the gold label is used for scoring.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
accuracy = (correct / len(gold_answers)) * 100

Common pitfalls

• Rule-based heuristics (e.g., picking the suspect with the longest chapter or most mentioned location) fail near random chance, so models cannot rely on simple keyword counting.
• Adding more few-shot examples does not always improve performance and can hit token limits or cause diminishing returns.
• Neurosymbolic methods struggle with entity extraction in natural language compared to templated datasets, as actions/observations span multiple paragraphs rather than isolated sentences.

Evidence (verbatim from paper)

Table 5 shows results over our LLMs with the CoT+ prompt as well as human performance. Llama 2 and Vicuna-based language models are able to get above chance for each domain but only slightly. ... GPT-4 performs the best out of all the models we tested, but still underperforms compared to humans.

Citation

@misc{sprague2023mus,
  title={MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning},
  author={Sprague et al. (2023)},
  year={2023},
  note={arXiv:2310.16049}
}

• arXiv: 2310.16049