Knowledgeberg Eval

knowledgeberg-eval

KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models — Zhang et al. (2026) (arXiv:2604.17621, 2026)

What this evaluates

Evaluates large language models' ability to systematically cover bounded knowledge universes and perform compositional set-based reasoning. It probes three failure stages: completeness (missing knowledge), awareness (failure to identify requirements), and application (incorrect execution) across multiple domains and languages.

Datasets

• KnowledgeBerg — total 4800; splits: test (-1)

Metrics

• Universe F1 (primary) — range: [0, 1]
  • Set-level F1 score computed as 2 * (Precision * Recall) / (Precision + Recall), where Precision = m / |P| and Recall = m / |G|. m is the number of one-to-one matched pairs between predicted and gold items after rule-based and LLM-as-judge matching.
• Accuracy — range: [0, 1]
  • Standard exact-match accuracy for multiple-choice knowledge-grounded reasoning questions (KRQs).

Input / output format

Input: For KRQs: a multiple-choice question requiring knowledge-grounded reasoning. For EQs: a question requiring enumeration of items from a bounded knowledge universe.

Output: For KRQs: a single selected option. For EQs: a delimited list of enumerated items (e.g., comma-separated).

Scoring recipe

gold_set = parse_and_normalize(gold_text)
pred_set = parse_and_normalize(pred_text)
matched_pairs = rule_match(gold_set, pred_set)
unmatched_pred = pred_set - matched_pairs.pred
judge = Qwen3_30B()
for p in unmatched_pred:
    if judge.is_equivalent(p, gold_set):
        matched_pairs.add(p)
m = len(matched_pairs)
precision = m / len(pred_set)
recall = m / len(gold_set)
f1 = 2 * precision * recall / (precision + recall)
return f1

Common pitfalls

• The LLM judge uses greedy one-to-one matching, which may fail to find the optimal bipartite matching between predictions and gold items.
• Rule-based parsing relies on common delimiters and may misparse complex list formats or nested enumerations.
• The judge operates with temperature=0.0, making it deterministic but potentially brittle on ambiguous or borderline cases.

Evidence (verbatim from paper)

We compute set-level precision, recall, and F1 as Precision = m / |P|, Recall = m / |G|, F1 = 2 * Precision * Recall / (Precision + Recall). We therefore adopt a hybrid protocol: rule-based matching first, followed by LLM judging only for unresolved cases.

Citation

@misc{zhang2026knowledgeberg,
  title={KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.17621}
}

• arXiv: 2604.17621