# Neurokg Eval

> Evaluates a heterogeneous graph transformer's ability to learn multi-scale biological relationships and predict missing links in a brain knowledge graph. It probes downstream capabilities including genome-wide screen enrichment, pesticide toxicity ranking, and drug repurposing forecasting across neurological diseases. Use when the user wants to benchmark on NeuroKG, or asks about evaluating this task. Reports AUROC.

- Skill: `qhjqhj00/neurokg-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/neurokg-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/neurokg-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/neurokg-eval

---


# neurokg-eval

> Graph AI generates neurological hypotheses validated in molecular, organoid, and clinical systems — Noori et al. (2025) (arXiv:2512.13724, 2025)

## What this evaluates

Evaluates a heterogeneous graph transformer's ability to learn multi-scale biological relationships and predict missing links in a brain knowledge graph. It probes downstream capabilities including genome-wide screen enrichment, pesticide toxicity ranking, and drug repurposing forecasting across neurological diseases.

## Datasets

- **NeuroKG** — total 147020; splits: held-out test (-1); repo https://github.com/mims-harvard/PROTON

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `Normalized Enrichment Score (NES)` — range: other
  - A Kolmogorov-Smirnov statistic adjusted for multiple hypothesis testing and gene set size, used in GSEA to quantify overrepresentation of experimental hits in predicted rankings.
- `macro-averaged recall at k (R@k)` — range: percent
  - The average percentage of known relevant drugs recovered within the top k predictions across all evaluated diseases.

## Input / output format

**Input**: Heterogeneous graph containing 147,020 nodes across 16 types (genes, proteins, cell types, brain regions, phenotypes, drugs) and 7,366,745 edges across 47 types. Tasks involve node pairs for link prediction, disease/compound nodes with graph context for downstream screens, and fine-tuning datasets of labeled pesticides.

**Output**: Continuous link prediction scores or ranks for node pairs; ranked lists of genes/proteins/compounds; or binary toxicity predictions for pesticides.

## Scoring recipe

```python
def compute_auroc(y_true, y_scores):
    return roc_auc_score(y_true, y_scores)

def compute_recall_at_k(relevant_items, ranked_list, k):
    top_k = ranked_list[:k]
    return len(set(relevant_items) & set(top_k)) / len(relevant_items)

def compute_nes(experimental_hits, predicted_rankings):
    return gsea_normalized_enrichment_score(experimental_hits, predicted_rankings)
```

## Common pitfalls

- Information leakage occurs if drug-disease edges for the target disease or related diseases are not removed from the graph before training disease-specific splits.
- Genes or proteins with direct links to the query node (e.g., PD or α-synuclein) must be excluded from enrichment analyses to prevent tautological results.
- List-size differences in GWAS hit sets must be controlled via random subsampling to ensure robust statistical comparisons across diseases.

## Evidence (verbatim from paper)

> Proton was trained on the NeuroKG dataset using a self-supervised link prediction objective (Methods Sec.[2.2]). Through Bayesian hyperparameter optimization, we selected a model architecture that achieved high link prediction performance (AUROC =0.9145; accuracy =82.23%) on the held-out test set.

## Citation

```bibtex
@misc{noori2025proton,
  title={Graph AI generates neurological hypotheses validated in molecular, organoid, and clinical systems},
  author={Noori et al. (2025)},
  year={2025},
  note={arXiv:2512.13724}
}
```

- arXiv: 2512.13724