Biomedqa Eval

biomedqa-eval

Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database — Mandarapu et al. (2026) (arXiv:2603.15080, 2026)

What this evaluates

Probes an AI agent's ability to answer pharmacology questions by querying federated biomedical knowledge graphs. It evaluates three access methods—direct MCP tools, text-to-Cypher generation, and standalone LLM reasoning—to measure factual accuracy and query efficiency.

Datasets

• BiomedQA — total 40; splits: test (40); repo https://github.com/samyama-ai/biomedqa

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total 40 pharmacology questions across 7 categories.

Input / output format

Input: 40 pharmacology questions spanning 7 categories, requiring answers derived from federated biomedical knowledge graphs (Pathways, Drug Interactions, Clinical Trials).

Output: Natural language answer or executed query result, depending on the evaluation approach (MCP tools, text-to-Cypher, or standalone LLM).

Scoring recipe

correct = 0
for question, prediction, gold in zip(questions, predictions, gold_answers):
    if prediction == gold:
        correct += 1
accuracy = (correct / len(questions)) * 100

Common pitfalls

• Schema hallucinations in text-to-Cypher approaches (e.g., non-existent edge traversals) can cause failures unrelated to factual knowledge.
• Token counting conventions differ across approaches; the NLQ endpoint handles LLM calls server-side, resulting in 0 reported tokens for that method.
• Latency varies drastically between single-KG lookups (~80-100 ms) and multi-hop cross-KG joins (3-4 s), so benchmarking conditions must be explicitly stated.

Evidence (verbatim from paper)

We introduce BiomedQA, a benchmark of 40 pharmacology questions across 7 categories over the three federated KGs. We compare three approaches: domain-specific MCP tools (parameterized Cypher templates), text-to-Cypher via the schema-aware NLQ endpoint (GPT-4o with full schema system prompt and few-shot examples), and standalone GPT-4o (no database access). Table 10: BiomedQA results (40 questions, 7.9M nodes, 3 federated KGs).

| Approach | Accuracy | Avg Latency | Avg Tokens |

Citation

@misc{mandarapu2026samyama,
  title={Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database},
  author={Mandarapu et al. (2026)},
  year={2026},
  note={arXiv:2603.15080}
}

• arXiv: 2603.15080