Pkgt Kg Construction Eval

pkgt-kg-construction-eval

An Open-Source Knowledge Graph Ecosystem for the Life Sciences — Callahan et al. (2023) (arXiv:2307.05727, 2023)

What this evaluates

Evaluates the PheKnowLator ecosystem's software features and computational performance against other biomedical KG construction tools, and measures construction efficiency on 12 benchmark knowledge graphs.

Datasets

• PKT Human Disease benchmark KGs — total ?; splits: test (12); repo https://github.com/callahantiff/PheKnowLator

Metrics

• coverage score (primary) — range: other
  • Average score across five assessment criteria (functionality, maturity, availability, usability, reproducibility) based on a 44-question survey.
• construction time — range: other
  • Wall-clock time in minutes to complete each of the three KG construction steps (Data Download, Edge List Creation, Graph Construction).
• memory usage — range: other
  • Peak memory consumption in GB during each construction step.

Input / output format

Input: Survey questionnaires for software comparison; raw biomedical data sources (ontologies, LOD datasets, molecular experiments) for KG construction.

Output: Survey scores per criterion; constructed knowledge graphs with node/edge counts, density, and self-loop statistics; time and memory metrics per step.

Scoring recipe

# Survey scoring
criteria = ["functionality", "maturity", "availability", "usability", "reproducibility"]
coverage_score = mean([survey_score(method, c) for c in criteria])

# Performance scoring
for step in ["Data Download", "Edge List Creation", "Graph Construction"]:
    time_min = measure_wall_clock_time(step)
    mem_gb = measure_peak_memory(step)
    record(step, time_min, mem_gb)

Common pitfalls

• Survey scores rely on self-reported GitHub metadata and may not reflect actual runtime performance or code quality.
• Construction time and memory metrics are hardware/environment dependent and not normalized across different test runs.

Evidence (verbatim from paper)

The average coverage score of the five assessment criteria was 3.93 (min=2.79, max=4.90). ... Performance metrics by KG construction step for each of the 12 PKT Human Disease benchmark KGs are shown in Supplementary Figure 2. On average, Step 1 (Data Download) took 2.30 minutes (1.80-3.72 minutes) and used an average of 7.93 GB of memory (7.86-7.99 GB).

Citation

@misc{callahan2023pkgt,
  title={An Open-Source Knowledge Graph Ecosystem for the Life Sciences},
  author={Callahan et al. (2023)},
  year={2023},
  note={arXiv:2307.05727}
}

• arXiv: 2307.05727