Graphwalker Eval

graphwalker-eval

GraphWalker: Graph-Guided In-Context Learning for Clinical Reasoning on Electronic Health Records — Yue Fang et al. (2026) (arXiv:2604.06684, 2026)

What this evaluates

Evaluates a graph-guided in-context learning framework for clinical reasoning on electronic health records. It probes the model's ability to select non-redundant, interacting demonstrations based on patient semantic graphs and information gain signals to improve prediction accuracy on mortality, length-of-stay, readmission, and clinical QA tasks.

Datasets

• MIMIC-III — total ?; splits: train (-1), val (-1), test (-1)
• MIMIC-IV — total ?; splits: train (-1), val (-1), test (-1)
• CMB — total ?; splits: test (-1)
• MedQA — total ?; splits: test (-1)
• CMB-clin — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the trade-off between true positive rate and false positive rate across classification thresholds.
• AUPRC (primary) — range: [0, 1]
  • Area under the precision-recall curve. Evaluates model performance on imbalanced datasets by plotting precision against recall.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• ma-ROC — range: [0, 1]
  • Macro-averaged ROC for discretized length-of-stay bins. Computes ROC per bin and averages them.
• mi-ROC — range: [0, 1]
  • Micro-averaged ROC for discretized length-of-stay bins. Aggregates contributions of all bins before computing ROC.
• exact-match (EM) — range: [0, 1]
  • Binary metric: 1 if the predicted answer exactly matches the ground truth, 0 otherwise. Averaged over the dataset.
• BLEU-1 / BLEU-4 — range: [0, 100]
  • N-gram precision metric (1-gram and 4-gram) with brevity penalty, standard for text generation evaluation.
• ROUGE-L — range: [0, 100]
  • Recall-oriented overlap metric based on the longest common subsequence between predicted and reference text.

Input / output format

Input: Patient-level EHR sequences (visit-sequenced) for prediction tasks; clinical questions/prompts for CMB, MedQA, and CMB-clin.

Output: Binary or categorical labels (e.g., mortality, readmission), discretized LOS bins, or free-text answers for clinical QA.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task in ['mortality', 'readmission']:
        return roc_auc_score(golds, predictions), \
               average_precision_score(golds, predictions), \
               f1_score(golds, predictions)
    elif task == 'los':
        bins = discretize(golds)
        return macro_roc_auc(bins, predictions), micro_roc_auc(bins, predictions)
    elif task in ['CMB', 'MedQA']:
        return np.mean([p == g for p, g in zip(predictions, golds)])
    elif task == 'CMB-clin':
        return bleu_1(predictions, golds), bleu_4(predictions, golds), rouge_l(predictions, golds)

Common pitfalls

• Discretizing length-of-stay (LOS) labels into bins changes the evaluation distribution and requires careful bin definition to match established benchmarks.
• Using generic semantic embeddings instead of EHR-specific encoders for demonstration retrieval leads to significant performance drops due to lack of clinical nuance.
• Ignoring redundancy and interaction effects among demonstrations causes diminishing marginal returns in context-constrained settings.

Evidence (verbatim from paper)

For mortality and readmission prediction, we employ three widely used evaluation metrics, including the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), and the F1-score. For LOS prediction, following Harutyunyan et al. (Harutyunyan et al., 2019), we discretize LOS labels into multiple bins and evaluate performance using macro-averaged ROC (ma-ROC) and micro-averaged ROC (mi-ROC).

Citation

@misc{fang2026graphwalker,
  title={GraphWalker: Graph-Guided In-Context Learning for Clinical Reasoning on Electronic Health Records},
  author={Yue Fang et al. (2026)},
  year={2026},
  note={arXiv:2604.06684}
}

• arXiv: 2604.06684