qhjqhj00 nanoknow-eval
NanoKnow: How to Know What Your Language Model Knows — Gu et al. (2026) (arXiv:2602.20122, 2026)
What this evaluates
Evaluates how pre-training data exposure and external context influence closed-book and open-book question answering accuracy. It probes the model's reliance on parametric knowledge versus retrieved evidence, and measures the impact of answer frequency and distractors.
Datasets
- Natural Questions — total ?; splits: supported (-1), unsupported (-1)
- SQuAD — total ?; splits: supported (-1), unsupported (-1)
Metrics
Exact Match (EM)(primary) — range: [0, 1]- Checks if any predefined correct answer string appears verbatim in the model's generated output. Returns 1 if matched, 0 otherwise.
LLM-Judge— range: [0, 1]- Uses Qwen3-14B to classify the model's output as correct or incorrect against predefined correct answers based on a prompt.
Input / output format
Input: Question text, optionally accompanied by a context passage (oracle FineWeb-Edu snippet or original SQuAD context).
Output: Free-form text answer generated by the model.
Scoring recipe
def score_exact_match(predictions, gold_answers):
return 1.0 if any(ans in pred for ans in gold_answers) else 0.0
def score_llm_judge(predictions, gold_answers, questions):
# Uses Qwen3-14B to classify correctness
prompt = f'Q: {q}\nPred: {pred}\nGold: {gold_answers}\nCorrect? (Yes/No)'
return 1.0 if qwen3_14b(prompt) == 'Yes' else 0.0
Common pitfalls
- Exact match penalizes semantically correct but paraphrased answers, potentially underestimating true capability.
- LLM-Judge results are contingent on Qwen3-14B's specific prompt and behavior, which may introduce judge bias or inconsistency.
- Context is strictly limited to a ~200-word window around the answer, which may truncate relevant supporting information.
Evidence (verbatim from paper)
To evaluate the accuracy of responses generated by nanochat, we use two approaches. The first is an exact match (EM) evaluation which checks if any of the predefined correct answers exactly appear in the model’s output. If there is a match, the answer is deemed correct; otherwise the answer is deemed incorrect. The next method we consider is an LLM-Judge, which given nanochat’s output and the predefined correct answers, classifies nanochat’s output as correct or not. For this, we leverage Qwen3-14B*(Yang et al., [2025])*.
Citation
@misc{gu2026nanoknow,
title={NanoKnow: How to Know What Your Language Model Knows},
author={Gu et al. (2026)},
year={2026},
note={arXiv:2602.20122}
}
- arXiv: 2602.20122