Knight Knave Eval

knight-knave-eval

On Memorization of Large Language Models in Logical Reasoning — Chulin Xie et al. (arXiv:2410.23123, 2024)

What this evaluates

Evaluates whether large language models rely on memorization versus genuine logical reasoning by measuring performance drops on logically equivalent but locally perturbed Knights and Knaves puzzles. It probes the model's ability to maintain consistent logical deductions when superficial or structural elements of the problem are altered.

Datasets

• Knights and Knaves (K&K) — total ?; splits: train (-1), test (-1)

Metrics

• LiMem (primary) — range: [0, 1]
  • LiMem measures memorization via performance inconsistency. It is computed as the difference between accuracy on original puzzles and accuracy on locally perturbed puzzles (leaf, statement, or language-level). A higher positive difference indicates stronger reliance on memorization rather than reasoning.

Input / output format

Input: Natural language description of a Knights and Knaves puzzle specifying N inhabitants, their logical statements (e.g., 'X is a knight and Y is a knave'), and a question asking to identify each inhabitant's role.

Output: A structured list identifying each person's role, e.g., '(1) Oliver is a knight (2) Jacob is a knave'.

Scoring recipe

def compute_limem(preds_orig, gold_orig, preds_pert, gold_pert):
    acc_orig = sum(1 for p, g in zip(preds_orig, gold_orig) if p == g) / len(gold_orig)
    acc_pert = sum(1 for p, g in zip(preds_pert, gold_pert) if p == g) / len(gold_pert)
    return acc_orig - acc_pert

Common pitfalls

• Perturbations must guarantee a unique solution different from the original; leaf perturbations may fail to find valid puzzles within the 2000 attempt limit.
• Language-level perturbations like 'flip role' (knave tells truth) exploit model biases and cause high error rates despite logically valid puzzles.
• The benchmark is dynamically generated with specific constraints (N=2-8, W=2, D=2), so results are not directly comparable to static benchmarks.

Evidence (verbatim from paper)

The paper introduces a novel memorization metric, LiMem, based on performance inconsistency under local perturbations in logical reasoning tasks, using a dynamically generated Knights and Knaves (K&K) puzzle benchmark. To support memorization measurement, the K&K Puzzle Perturber generate an perturbed version of a given puzzle. We design the perturbation with the following considerations: The perturbation should be “local”, in the sense that the perturbed problem should be similar to the original problem when measured in some superficial ways, such as edit distance. The perturbation should keep the nature of the problem, i.e., the perturbed problem should be solvable using the same underlying principle, and the difficulty level should be roughly maintained. The perturbed puzzle should have a unique solution, which should be different from the solution of the original puzzle.

Citation

@misc{xie2024memorization,
  title={On Memorization of Large Language Models in Logical Reasoning},
  author={Chulin Xie et al.},
  year={2024},
  note={arXiv:2410.23123}
}

• arXiv: 2410.23123