Representation Benchmark Eval

representation-benchmark-eval

Representation Before Training: A Fixed-Budget Benchmark for Generative Medical Event Models — Lee et al. (2026) (arXiv:2604.16775, 2026)

What this evaluates

Evaluates how input representation choices—quantization granularity, value encoding, temporal encoding, and vocabulary remapping—affect downstream predictive performance on clinical outcomes. It probes the model's ability to extract and utilize structured medical event sequences for binary classification and regression tasks.

Datasets

• MIMIC-IV — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• Spearman ρ — range: [-1, 1]
  • Spearman rank correlation coefficient; measures the monotonic relationship between predicted and actual continuous values based on rank ordering.

Input / output format

Input: Sequences of clinical events (labs, vitals, interventions) formatted with specific representation choices (quantization bins, value encoders, temporal embeddings, and vocabulary mappings).

Output: Binary classification probabilities for outcomes (e.g., mortality, LOS > 7d, ICU admission) and continuous regression predictions for outcomes (e.g., LOS in hours, peak/min laboratory values).

Scoring recipe

def compute_auroc(preds, golds):
    return roc_auc_score(golds, preds)

def compute_spearman(preds, golds):
    return spearmanr(golds, preds).correlation

def evaluate(predictions, golds, task_type):
    if task_type == 'binary':
        return compute_auroc(predictions, golds)
    return compute_spearman(predictions, golds)

Common pitfalls

• Assuming finer quantization granularity (e.g., trentiles vs. deciles) universally improves performance; gains are highly outcome-specific.
• Comparing fused vs. unfused tokenization without accounting for vocabulary size and embedding parameter differences.
• Ignoring temporal outcome dependencies; e.g., vitals are more tightly coupled to admission timelines than laboratory extrema.
• Failing to report bootstrap confidence intervals and apply multiple testing corrections (BH-adjusted p-values) when claiming statistical significance.

Evidence (verbatim from paper)

Across the three tested temporal encodings, inserting time tokens was the weakest option on average. Within the discrete family, the event-order-only configuration (discrete_none) outperformed the time token configuration (discrete_tt) on ICU admission (AUROC 0.816 [0.808, 0.824] vs. 0.806 [0.797, 0.814], Δ=−0.0106 [−0.0163, −0.0048], BH-adjusted p=0.0015) and hospital length of stay >7 days (0.789 [0.785, 0.792] vs. 0.785 [0.781, 0.789], Δ=−0.0038 [−0.0063, −0.0016], BH-adjusted p=0.0029).

Citation

@misc{lee2026representation,
  title={Representation Before Training: A Fixed-Budget Benchmark for Generative Medical Event Models},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2604.16775}
}

• arXiv: 2604.16775