Validation Strategy

Validation Strategy

Overview

Validation ensures that findings are reliable and generalizable. The approach differs between predictive modeling (ML) and statistical inference, though both require rigor.

Machine Learning Validation

Core Principle

Training performance means nothing. Only held-out performance matters.

Data Splitting Strategies

Simple Train/Test Split

Full Data
├── Training (70-80%) → fit model
└── Test (20-30%) → evaluate final model (touch ONCE)

• Pros: Simple, fast
• Cons: Single estimate, variance in performance
• Use when: Large dataset, quick baseline

Train/Validation/Test Split

Full Data
├── Training (60%) → fit model
├── Validation (20%) → tune hyperparameters, select model
└── Test (20%) → final evaluation (touch ONCE)

• Pros: Separates tuning from evaluation
• Cons: Less training data, still single test estimate
• Use when: Hyperparameter tuning needed

K-Fold Cross-Validation

Full Data split into K folds
Repeat K times:
  - Train on K-1 folds
  - Validate on 1 fold
Report mean ± SD across folds

• Pros: Uses all data, variance estimate
• Cons: Computationally expensive, no single "final" model
• Use when: Moderate sample size, want uncertainty

Nested Cross-Validation

Outer loop: K-fold for performance estimation
  Inner loop: K-fold for hyperparameter tuning

• Pros: Unbiased performance estimate with tuning
• Cons: Very computationally expensive
• Use when: Hyperparameters must be tuned, rigorous reporting needed

Critical: Patient-Level Splitting

NEVER split at the admission or ICU-stay level if patients have multiple records.

Wrong:

Admission 1 (Patient A) → Train
Admission 2 (Patient A) → Test  ← DATA LEAKAGE!

Correct:

All of Patient A's admissions → Train
All of Patient B's admissions → Test

Implementation:

from sklearn.model_selection import GroupKFold

# 'subject_id' ensures all admissions from same patient stay together
gkf = GroupKFold(n_splits=5)
for train_idx, test_idx in gkf.split(X, y, groups=df['subject_id']):
    ...

Temporal Validation

For EHR data, temporal splits often better reflect real-world deployment:

Data from 2008-2016 → Train
Data from 2017-2019 → Test

Why:

• Simulates prospective deployment
• Accounts for temporal drift (practice changes, ICD-10 transition)
• More honest estimate of future performance

Multi-Site Validation (eICU)

For multi-center data, consider site-level splits:

Hospitals A, B, C → Train
Hospitals D, E → Test

Why: Tests generalizability across institutions

Metrics for Binary Outcomes

Discrimination vs. Calibration

• Discrimination: Can the model rank patients? (Higher risk → higher score)
• Calibration: Are predicted probabilities accurate? (30% prediction → 30% actually have event)

Both matter for clinical use. A well-discriminating but poorly calibrated model gives misleading risk estimates.

Calibration Assessment

1. Calibration plot: Predicted vs observed, should follow 45° line
2. Hosmer-Lemeshow test: Formal test (but sensitive to sample size)
3. Calibration slope: Should be ~1.0 (if <1, overfitting; if >1, underfitting)

Handling Class Imbalance

Warning: SMOTE and oversampling can leak information if done before splitting. Always apply within each training fold, never to full data.

Bootstrap for Confidence Intervals

Repeat B times (e.g., B=1000):
  1. Sample with replacement from test set
  2. Calculate metric on bootstrap sample
Report: median and 2.5th/97.5th percentiles

Gives confidence intervals for AUROC, AUPRC, etc.

Statistical Inference Validation

Confidence Intervals

Always report CIs alongside point estimates.

CIs are more informative than p-values alone.

Significance Testing Framework

Multiple Comparisons

Problem: Testing 20 hypotheses at α=0.05 expects 1 false positive by chance.

Effect Sizes

P-values alone are insufficient. Report effect sizes.

Power and Sample Size

Pre-Study Power Analysis

Before starting:

• Define minimum clinically important effect
• Calculate required n for 80% or 90% power

Post-Study: Sensitivity, Not Power

Do NOT compute post-hoc power (it's a function of p-value, tells nothing new).

Instead, compute detectable effect size:

• "With n=500 and α=0.05, we had 80% power to detect OR ≥ 1.3"

Sensitivity Analyses

Planned analyses to test robustness:

Reporting Checklist

For Predictive Models

• Split strategy clearly described (patient-level, temporal, etc.)
• Cross-validation or held-out test used
• Discrimination metrics (AUROC, AUPRC)
• Calibration assessed (plot, slope)
• Confidence intervals via bootstrap
• Class imbalance handling described
• Feature selection done within CV (no leakage)

For Statistical Inference

• Primary outcome pre-specified
• Sample size justification
• Effect size with confidence interval
• P-value and α level stated
• Multiple comparison correction (if applicable)
• Sensitivity analyses performed
• Assumptions checked and reported

Common Validation Mistakes

1. Reporting training AUROC: Always use held-out data

2. Admission-level splits with repeated patients: Data leakage

3. Tuning on test set: Use separate validation set

4. SMOTE before splitting: Information leakage

5. Only reporting p-values: Effect size and CI needed

6. Post-hoc power analysis: Uninformative; use sensitivity analysis

7. Single train/test split with small data: High variance; use CV

8. Ignoring calibration: Discrimination alone insufficient for clinical use