Biomedical Cypher Eval

biomedical-cypher-eval

Combining LLMs and Knowledge Graphs to Reduce Hallucinations in Question Answering — Pusch et al. (2024) (arXiv:2409.04181, 2024)

What this evaluates

Probes an LLM's ability to generate syntactically and semantically correct Cypher queries for a biomedical knowledge graph, and execute them to answer domain-specific questions without hallucination.

Datasets

• Custom Biomedical QA Benchmark — total 50; splits: test (50)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of questions where the generated Cypher query, when executed against the knowledge graph, returns exactly the manually verified expected result. Calculated as (correct answers / 50).

Input / output format

Input: A biomedical question requiring knowledge graph retrieval, provided alongside the relevant KG schema or retrieved context for Cypher generation.

Output: A single Cypher query string intended to retrieve the answer from the knowledge graph.

Scoring recipe

correct = 0
for question, expected_result in dataset:
    generated_query = model.generate(question)
    actual_result = execute_cypher(generated_query, kg)
    if actual_result == expected_result:
        correct += 1
accuracy = correct / len(dataset)

Common pitfalls

• Execution must exactly match the manually verified expected result; semantic similarity or partial matches do not count as correct.
• The dataset is small (50 questions) and domain-specific, so results may not generalize to other knowledge graphs or question types.
• Query correctness depends on both syntactic validity and semantic alignment with the KG schema, not just LLM text generation quality.

Evidence (verbatim from paper)

We evaluated each LLM based on the number of correct answers across all 50 questions in our dataset (see [3.2]). A question was considered to be correctly answered if the resulting Cypher query returns the expected result from the Knowledge Graph. The correct result has been created manually in the data generation process.

Citation

@misc{pusch2024combining,
  title={Combining LLMs and Knowledge Graphs to Reduce Hallucinations in Question Answering},
  author={Pusch et al. (2024)},
  year={2024},
  note={arXiv:2409.04181}
}

• arXiv: 2409.04181