Medg Krp Eval

medg-krp-eval

MedG-KRP: Medical Graph Knowledge Representation Probing — Rosenbaum et al. (2024) (arXiv:2412.10982, 2024)

What this evaluates

Evaluates large language models' ability to generate causal knowledge graphs from single medical concepts. It probes biomedical reasoning, causal understanding, and factual consistency by comparing generated graphs against human expert judgments and a ground-truth biomedical ontology (BIOS).

Datasets

• MedG-KRP — total 20; splits: test (20); repo https://github.com/nyuolab/MedG-KRP

Metrics

• Human Accuracy — range: [1, 4]
  • Average score (1-4 scale) from three medical student reviewers assessing the medical correctness of all concepts, relationships, and implied causal pathways in the generated graph.
• Human Comprehensiveness — range: [1, 4]
  • Average score (1-4 scale) from three reviewers assessing whether the graph covers all necessary medical concepts for a proper understanding of the given disease.
• Precision (primary) — range: [0, 1]
  • n_hit / |E_g|, where n_hit is the count of generated edges with a shortest path ≤ 7 in the BIOS ground-truth graph, and |E_g| is the total number of generated edges.
• Recall — range: [0, 1]
  • n_hit / |E_rel|, where |E_rel| is the set of BIOS edges connected to any node in the generated graph, and n_hit is defined as above.

Input / output format

Input: A single medical condition (root concept) passed through zero-shot system, expansion, and edge-check prompts. The model iteratively generates nodes and edges based on causal relationships (causes/caused-by) using counterfactual reasoning.

Output: A causal knowledge graph represented as a set of nodes and directed edges, generated iteratively via prompt responses.

Scoring recipe

def score_graph(generated_graph, bios_graph):
    # Node mapping handled externally via e5 embeddings + GPT-4
    generated_nodes = set(generated_graph.nodes)
    generated_edges = list(generated_graph.edges)
    
    hits = 0
    for u, v in generated_edges:
        if shortest_path_length(u, v, bios_graph) <= 7:
            hits += 1
            
    precision = hits / len(generated_edges) if generated_edges else 0
    
    relevant_bios_edges = [
        e for e in bios_graph.edges 
        if e[0] in generated_nodes or e[1] in generated_nodes
    ]
    recall = hits / len(relevant_bios_edges) if relevant_bios_edges else 0
    
    return precision, recall

Common pitfalls

• Using a path length threshold of ≤ 7 instead of direct edge matching may overestimate precision/recall for indirect medical relationships.
• Node mapping relies on e5 embeddings plus GPT-4 verification, which can produce false matches or miss valid mappings if LLM outputs deviate from BIOS terminology.
• Human evaluation uses medical students rather than board-certified physicians, potentially limiting clinical validity despite physician-verified root concepts.

Evidence (verbatim from paper)

We calculated the precision and recall of generated edges using LABEL:alg:prec_recall. For each generated graph, we iterate through all edges and check if there is a path of length less than or equal to 7 between the two corresponding concepts in the ground truth. This means, for a given edge, the number of intermediary nodes in the ground truth between the two nodes that constitute the edge must be less than or equal to five. If a path in the ground truth satisfies this condition, it is marked as a hit. Otherwise, it is marked as a miss.

Citation

@misc{rosenbaum2024medgkrp,
  title={MedG-KRP: Medical Graph Knowledge Representation Probing},
  author={Rosenbaum et al. (2024)},
  year={2024},
  note={arXiv:2412.10982}
}

• arXiv: 2412.10982