Ko Musr Eval

ko-musr-eval

Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean — Park et al. (2025) (arXiv:2510.24150, 2025)

What this evaluates

This benchmark evaluates multistep soft reasoning capabilities of LLMs in long narratives, specifically testing logical deduction, object placement tracking, and team allocation across English and Korean languages. It probes cross-lingual reasoning transfer and the impact of in-context learning strategies like Chain-of-Thought prompting and task-specific hints.

Datasets

• Ko-MuSR — total ?; splits: test (-1); repo https://github.com/mcrl/Ko-MuSR
• MuSR — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of instances) × 100. Evaluated per subtask (MM, OP, TA) and averaged across two independent runs.

Input / output format

Input: Long narrative passages (English or Korean) followed by a multiple-choice question. Prompts may include 0, 1, or 3 in-context examples, Chain-of-Thought reasoning traces, and task-specific hints (4-5 sentences).

Output: A single selected multiple-choice answer corresponding to the question.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
# Evaluated twice; final score is the average of the two runs.

Common pitfalls

• Random baseline accuracy varies by subtask due to differing numbers of answer choices (50% for MM, 24.6% for OP, 33.3% for TA).
• Applying advanced prompting (3-shot CoT + hints) can significantly degrade performance on smaller language models (SLMs), making configuration choice critical for fair comparison.

Evidence (verbatim from paper)

We measure model accuracy for each subtask, Murder Mysteries (MM), Object Placements (OP), and Team Allocations (TA). Some MuSR instances for object placement contain five answer choices, which leads to a random baseline accuracy of 24.6. For Ko-MuSR, we ensure that all questions have a consistent number of answer choices across subtasks. We clearly specify the number of examples, whether Chain-of-Thought (CoT) reasoning is used by referring to it as CoT or Direct for straightforward responses, and we indicate the presence or absence of hints by using Hint and No Hint.

Citation

@misc{park2025ko_musr,
  title={Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2510.24150}
}

• arXiv: 2510.24150