Mine Kg Eval

mine-kg-eval

SocraticKG: Knowledge Graph Construction via QA-Driven Fact Extraction — Choi et al. (2026) (arXiv:2601.10003, 2026)

What this evaluates

Evaluates the ability of LLM-based methods to construct knowledge graphs from raw text that preserve factual information and maintain structural coherence. It measures how well extracted graphs retain ground-truth atomic facts and how densely connected and non-fragmented the resulting graphs are.

Datasets

• MINE — total 1500; splits: test (1500)

Metrics

• Factual Retention Score (primary) — range: percent
  • Percentage of ground-truth facts logically supported by a retrieved local subgraph (top-8 semantically similar nodes + 2-hop neighbors) as judged by an LLM (GPT-4o).
• Average Degree — range: other
  • Average number of unique neighboring nodes per node, computed as 2E/N where E is edges and N is nodes.
• Triple Count — range: other
  • Total number of atomic facts (triples) extracted and externalized in the graph.
• Normalized Fragmentation Index — range: [0, 1]
  • Measures graph fragmentation as (C-1)/(N-1) where C is connected components and N is nodes. Ranges from 0 (fully connected) to 1 (completely fragmented).

Input / output format

Input: Raw text articles (100 diverse documents).

Output: Knowledge graphs represented as sets of extracted triples/entities per article.

Scoring recipe

supported_count = 0
for fact in ground_truth_facts:
    subgraph = retrieve_top_k_nodes(fact, k=8) + get_2hop_neighbors(subgraph)
    if llm_judge(subgraph_context, fact).is_supported:
        supported_count += 1
factual_retention_score = (supported_count / len(ground_truth_facts)) * 100

nfi = (num_connected_components - 1) / (num_nodes - 1)

Common pitfalls

• LLM-as-a-judge verification may introduce bias or inconsistency despite temperature=0.
• Retrieving only top-8 similar nodes plus 2-hop neighbors can miss distant but logically relevant facts, artificially lowering retention scores.
• Graph structural metrics (Deg, NFI) depend heavily on the canonicalization and entity clustering steps (max 128 elements per cluster), which vary across methods.

Evidence (verbatim from paper)

Factual Retention Score

As the primary metric, we measured the proportion of ground-truth facts successfully recovered from the constructed KGs. Following the MINE benchmark protocol, we retrieved a local subgraph for each fact, consisting of the top-8 nodes most semantically similar to the target statement and their 2-hop neighbors. An LLM-judge then determined whether the fact was logically supported by the retrieved subgraph context. The score represents the percentage of verifiable facts, reflecting how well the graph preserves information from the source text for downstream tasks such as retrieval and reasoning.

Citation

@misc{choi2026socratickg,
  title={SocraticKG: Knowledge Graph Construction via QA-Driven Fact Extraction},
  author={Choi et al. (2026)},
  year={2026},
  note={arXiv:2601.10003}
}

• arXiv: 2601.10003