# Knowledgeberg Eval

> knowledgeberg-eval

- Skill: `qhjqhj00/knowledgeberg-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/knowledgeberg-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/knowledgeberg-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/knowledgeberg-eval

---


# knowledgeberg-eval

> KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models — Zhang et al. (2026) (arXiv:2604.17621, 2026)

## What this evaluates

Evaluates large language models' ability to systematically cover bounded knowledge universes and perform compositional set-based reasoning. It probes three failure stages: completeness (missing knowledge), awareness (failure to identify requirements), and application (incorrect execution) across multiple domains and languages.

## Datasets

- **KnowledgeBerg** — total 4800; splits: test (-1)

## Metrics

- `Universe F1` **(primary)** — range: [0, 1]
  - Set-level F1 score computed as 2 * (Precision * Recall) / (Precision + Recall), where Precision = m / |P| and Recall = m / |G|. m is the number of one-to-one matched pairs between predicted and gold items after rule-based and LLM-as-judge matching.
- `Accuracy` — range: [0, 1]
  - Standard exact-match accuracy for multiple-choice knowledge-grounded reasoning questions (KRQs).

## Input / output format

**Input**: For KRQs: a multiple-choice question requiring knowledge-grounded reasoning. For EQs: a question requiring enumeration of items from a bounded knowledge universe.

**Output**: For KRQs: a single selected option. For EQs: a delimited list of enumerated items (e.g., comma-separated).

## Scoring recipe

```python
gold_set = parse_and_normalize(gold_text)
pred_set = parse_and_normalize(pred_text)
matched_pairs = rule_match(gold_set, pred_set)
unmatched_pred = pred_set - matched_pairs.pred
judge = Qwen3_30B()
for p in unmatched_pred:
    if judge.is_equivalent(p, gold_set):
        matched_pairs.add(p)
m = len(matched_pairs)
precision = m / len(pred_set)
recall = m / len(gold_set)
f1 = 2 * precision * recall / (precision + recall)
return f1
```

## Common pitfalls

- The LLM judge uses greedy one-to-one matching, which may fail to find the optimal bipartite matching between predictions and gold items.
- Rule-based parsing relies on common delimiters and may misparse complex list formats or nested enumerations.
- The judge operates with temperature=0.0, making it deterministic but potentially brittle on ambiguous or borderline cases.

## Evidence (verbatim from paper)

> We compute set-level precision, recall, and F1 as Precision = m / |P|, Recall = m / |G|, F1 = 2 * Precision * Recall / (Precision + Recall). We therefore adopt a hybrid protocol: rule-based matching first, followed by LLM judging only for unresolved cases.

## Citation

```bibtex
@misc{zhang2026knowledgeberg,
  title={KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.17621}
}
```

- arXiv: 2604.17621