Medbert Disease Prediction Eval

medbert-disease-prediction-eval

Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction — Rasmy et al. (2020) (arXiv:2005.12833, 2020)

What this evaluates

Evaluates disease prediction performance using pre-trained contextualized embeddings on structured electronic health records. It probes the model's ability to capture temporal dependencies and long-term patient history from ICD-coded visit sequences, particularly in low-data transfer-learning scenarios.

Datasets

• DHF-Cerner — total ?; splits: train (-1), test (-1)
• PaCa-Cerner — total ?; splits: train (-1), test (-1)
• PaCa-Truven — total ?; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Computed by plotting the true positive rate against the false positive rate at various classification thresholds and calculating the area under the curve. Reported as mean ± standard deviation across evaluation runs.

Input / output format

Input: Temporal sequences of structured EHR data containing ICD-9/10 diagnostic codes across multiple patient visits, optionally augmented with pre-trained contextual embeddings.

Output: Binary or multi-class disease prediction probability/label.

Scoring recipe

def compute_auc(predictions, gold_labels):
    fpr, tpr, _ = roc_curve(gold_labels, predictions)
    return auc(fpr, tpr)

# Average across test splits, report mean and std
def evaluate(dataset_preds, dataset_labels):
    scores = [compute_auc(p, l) for p, l in zip(dataset_preds, dataset_labels)]
    return mean(scores), std(scores)

Common pitfalls

• AUC values are reported inconsistently in the text as both percentages (e.g., 'over 80 on AUC') and decimals (e.g., '0.65', '0.75'), which can cause confusion during replication.
• Small training sizes (e.g., 100-500 samples) yield heavily overlapping standard deviations, making it difficult to claim statistical significance between models without proper hypothesis testing.
• The evaluation mixes base models (GRU, Bi-GRU, RETAIN) with and without Med-BERT embeddings, requiring careful isolation of the embedding contribution versus architectural changes.

Evidence (verbatim from paper)

Table 3 presents the AUCs for Ex-1 on the three evaluation tasks. ... Table 3. Average AUC values and standard deviations for the different methods for the three evaluation tasks.

Citation

@misc{rasmy2020medbert,
  title={Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction},
  author={Rasmy et al. (2020)},
  year={2020},
  note={arXiv:2005.12833}
}

• arXiv: 2005.12833