# Kale Eval

> Evaluates large language models' ability to manipulate and apply stored knowledge across logical reasoning, reading comprehension, and natural language understanding tasks. It specifically probes the 'known & incorrect' phenomenon where models possess relevant facts but fail to apply them correctly during inference. Use when the user wants to benchmark on AbsR, Commonsense (Common), Big Bench Hard (BBH), RACE-H, RACE-M, MMLU, ARC-c, ARC-e, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/kale-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/kale-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/kale-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/kale-eval

---


# kale-eval

> KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learning — Qitan Lv et al. (arXiv:2601.07430, 2026)

## What this evaluates

Evaluates large language models' ability to manipulate and apply stored knowledge across logical reasoning, reading comprehension, and natural language understanding tasks. It specifically probes the 'known & incorrect' phenomenon where models possess relevant facts but fail to apply them correctly during inference.

## Datasets

- **AbsR** — total ?; splits: test (-1)
- **Commonsense (Common)** — total ?; splits: test (-1)
- **Big Bench Hard (BBH)** — total ?; splits: test (-1)
- **RACE-H** — total ?; splits: test (-1)
- **RACE-M** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **ARC-c** — total ?; splits: test (-1)
- **ARC-e** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard exact-match or multiple-choice accuracy: the proportion of test instances where the model's predicted answer exactly matches the ground truth label.

## Input / output format

**Input**: Standard benchmark prompts for logical reasoning, reading comprehension, and natural language understanding tasks.

**Output**: Model-generated answers or selected multiple-choice options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Benchmarks use different multiple-choice formats (e.g., ARC-c vs ARC-e, MMLU) which may require different parsing strategies for exact-match accuracy.
- The paper reports results on specific backbone models but does not specify prompt templates or few-shot settings used for each benchmark, which can significantly affect accuracy.
- Accuracy is reported as a percentage in tables but defined as a proportion in text; ensure consistent scaling when reproducing.

## Evidence (verbatim from paper)

> We use accuracy as the evaluation metric. More details and baseline descriptions are provided in Appendix [G].

## Citation

```bibtex
@misc{lv2026kale,
  title={KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learning},
  author={Qitan Lv et al.},
  year={2026},
  note={arXiv:2601.07430}
}
```

- arXiv: 2601.07430