Yokai Eval

yokai-eval

Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales — Tsutsumi et al. (2025) (arXiv:2506.03619, 2025)

What this evaluates

Assesses large language models' knowledge of Japanese yokai (folklore creatures) through multiple-choice questions. It probes cultural and linguistic familiarity with Japanese folklore, revealing how training data exposure and language affect cross-cultural knowledge retention.

Datasets

• YokaiEval — total 809; splits: test (809); repo https://github.com/CyberAgentAILab/YokaiEval

Metrics

• correctness (primary) — range: [0, 1]
  • Calculated as the proportion of questions where the GPT-4o evaluator outputs 'true' for the model's response. The evaluator checks if the model's output matches the correct choice, ignoring hedging language but penalizing multiple choices or no answer (null).

Input / output format

Input: A multiple-choice question about a Japanese yokai, the correct answer choice, and the model's generated response.

Output: The model must output a single choice from four options. The GPT-4o evaluator outputs 'true', 'false', or 'null'.

Scoring recipe

def compute_correctness(predictions, golds):
    true_count = 0
    for pred, gold in zip(predictions, golds):
        judge = gpt4o_evaluate(pred, gold)  # outputs 'true', 'false', or 'null'
        if judge == 'true':
            true_count += 1
    return true_count / len(predictions)

Common pitfalls

• The 1-shot prompt includes a specific example with hedging language ('educated guess') that must be replicated exactly to avoid biasing the GPT-4o evaluator.
• Responses with multiple choices or no answer are scored as 'null', which can artificially deflate correctness if not filtered or handled consistently.
• Evaluation relies on GPT-4o's semantic matching rather than exact string matching, making results sensitive to prompt formatting and model version.

Evidence (verbatim from paper)

Please determine the correctness of the AI assistant’s answer to the user's multiple-choice question displayed below. You need to evaluate the response strictly in the format of true, false, or null as shown in the output example. Output false if the response contains an incorrect choice. Output true if the response contains the correct choice. Output null if the response does not provide any answer.

Citation

@misc{tsutsumi2025yokai,
  title={Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales},
  author={Tsutsumi et al. (2025)},
  year={2025},
  note={arXiv:2506.03619}
}

• arXiv: 2506.03619