Kgquiz Eval

kgquiz-eval

KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models — Bai et al. (2023) (arXiv:2310.09725, 2023)

What this evaluates

Evaluates large language models' ability to store, retrieve, and reason over factual knowledge encoded in parametric memory across five progressively complex tasks. It probes basic fact verification, multiple-choice discrimination, open-ended entity generation, multi-hop factual editing, and comprehensive entity description generation. It measures how well models generalize encoded knowledge across commonsense, encyclopedic, and biomedical domains under increasing reasoning complexity.

Datasets

• KGQuiz — total ?; splits: test (-1); repo https://github.com/leopoldwhite/KGQuiz

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy for binary (True/False) and multiple-choice tasks. Calculated as the fraction of correctly predicted labels or options.
• LCS — range: [0, 1]
  • Longest Common Subsequence ratio: LCS = Len(s) / max(Len(t_o), Len(t)), where s is the longest common subsequence between model output t_o and ground truth t.
• F1-score — range: [0, 1]
  • Token-level F1: F1 = 2PR/(P+R), where P = |C|/|t_o| and R = |C|/|t_g|, with C being the set of common tokens.
• Semantic Match — range: [0, 1]
  • Binary metric based on InstructGPT Ada embeddings: SM = 1 if cosine_similarity(enc(t_o), enc(t)) >= θ, else 0. Threshold θ is calibrated on a held-out validation set.
• Precision — range: [0, 1]
  • For open-ended generation: Precision = |O ∩ G| / |O|, where O is the set of predicted fact triples and G is the ground truth triple set, filtered by Semantic Match = 1.
• Recall — range: [0, 1]
  • For open-ended generation: Recall = |O ∩ G| / |G|, using the same triple intersection logic as Precision.

Input / output format

Input: Varies by task: (1) 'Is the statement h r t True or False?', (2) Statement with [MASK] and m answer options, (3) Statement with [MASK], (4) Multi-hop knowledge path with one entity replaced by a negative sample, (5) 'Tell me some facts about h'.

Output: Varies by task: (1) 'True' or 'False', (2) Selected option, (3) Generated entity, (4) Corrected statement or revised entity, (5) Natural language text describing facts about h.

Scoring recipe

def score_kgquiz(pred, gold, task_type):
    if task_type in ['true_false', 'multiple_choice']:
        return 1.0 if pred.strip().lower() == gold.strip().lower() else 0.0
    elif task_type in ['blank_filling', 'factual_editing']:
        lcs_len = longest_common_subsequence(pred, gold)
        lcs = lcs_len / max(len(pred), len(gold))
        pred_tokens, gold_tokens = set(pred.split()), set(gold.split())
        common = pred_tokens & gold_tokens
        p = len(common) / len(pred_tokens) if pred_tokens else 0
        r = len(common) / len(gold_tokens) if gold_tokens else 0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        sm = 1.0 if cosine_similarity(embed(pred), embed(gold)) >= THETA else 0.0
        return {'LCS': lcs, 'F1': f1, 'SM': sm}
    elif task_type == 'open_ended':
        pred_triples = extract_triples(pred)
        gold_triples = extract_triples(gold)
        matched = [t for t in pred_triples if any(sm_score(t, gt) >= THETA for gt in gold_triples)]
        precision = len(matched) / len(pred_triples) if pred_triples else 0
        recall = len(matched) / len(gold_triples) if gold_triples else 0
        return {'Precision': precision, 'Recall': recall}

Common pitfalls

• Using exact string matching for generation/editing tasks instead of LCS, F1, or Semantic Match, which unfairly penalizes paraphrased or tokenized outputs.
• Applying a fixed threshold for the AdaScore semantic match instead of using the validation-set-calibrated θ specified in the paper.
• For factual editing, comparing the full generated string against the ground truth without first isolating the revised entity via longest common substring matching.

Evidence (verbatim from paper)

We denote the Longest Common Subsequence of $t_{o}$ and $t$ as $\boldsymbol{s}$, and $\mathrm{LCS}=\frac{\mathrm{Len}(\boldsymbol{s})}{\max{\mathrm{Len}(t_{o}),\mathrm{Len}(t)}}$ • F1-score: We denote the set of common tokens in both ${t_{o}}$ and ${t}$ as $C$. We denote the F1-score of $t_{o}$ and ${t}$ as $\mathrm{F1}=\frac{2PR}{P+R}$, where $P=\frac{|C|}{|{t_{o}}|}$,$R=\frac{|C|}{|{t_{g}}|}$. • Semantic Match: We measure semantic similarity between the model’s output and the correct answer using cosine similarity on embeddings obtained via InstructGPT Ada LLM $\mathrm{enc(\cdot)}$. This gives us the $\mathrm{AdaScore}(t_{o},t)=\mathrm{sim}(\mathrm{enc(t_{o})},\mathrm{enc(t)})$. A threshold $\theta$ of Adascore is based on a held-out validation set (detailed in Appendix[D]) to determine whether the model-generated answer and the ground truth are a semantically exact match. Concretely, we define the semantic match metric as SM$(t_{o},t)=1$ if $\mathrm{AdaScore}(t_{o},t)\geq\theta$, else 0.

Citation

@misc{bai2023kgquiz,
  title={KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models},
  author={Bai et al. (2023)},
  year={2023},
  note={arXiv:2310.09725}
}

• arXiv: 2310.09725