Kale Eval

kale-eval

KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learning — Qitan Lv et al. (arXiv:2601.07430, 2026)

What this evaluates

Evaluates large language models' ability to manipulate and apply stored knowledge across logical reasoning, reading comprehension, and natural language understanding tasks. It specifically probes the 'known & incorrect' phenomenon where models possess relevant facts but fail to apply them correctly during inference.

Datasets

• AbsR — total ?; splits: test (-1)
• Commonsense (Common) — total ?; splits: test (-1)
• Big Bench Hard (BBH) — total ?; splits: test (-1)
• RACE-H — total ?; splits: test (-1)
• RACE-M — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• ARC-c — total ?; splits: test (-1)
• ARC-e — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match or multiple-choice accuracy: the proportion of test instances where the model's predicted answer exactly matches the ground truth label.

Input / output format

Input: Standard benchmark prompts for logical reasoning, reading comprehension, and natural language understanding tasks.

Output: Model-generated answers or selected multiple-choice options.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Benchmarks use different multiple-choice formats (e.g., ARC-c vs ARC-e, MMLU) which may require different parsing strategies for exact-match accuracy.
• The paper reports results on specific backbone models but does not specify prompt templates or few-shot settings used for each benchmark, which can significantly affect accuracy.
• Accuracy is reported as a percentage in tables but defined as a proportion in text; ensure consistent scaling when reproducing.

Evidence (verbatim from paper)

We use accuracy as the evaluation metric. More details and baseline descriptions are provided in Appendix [G].

Citation

@misc{lv2026kale,
  title={KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learning},
  author={Qitan Lv et al.},
  year={2026},
  note={arXiv:2601.07430}
}

• arXiv: 2601.07430