Ukb Disease Risk Eval

ukb-disease-risk-eval

Multimodal LLMs for health grounded in individual-specific data — Belyaeva et al. (2023) (arXiv:2307.09018, 2023)

What this evaluates

Evaluates the ability of multimodal LLMs to predict binary disease risk from individual-specific clinical data, including tabular features and time-series spirograms. It tests how well serialized text and cross-modal embeddings integrate to produce accurate risk scores for conditions like asthma and diabetes.

Datasets

• UK Biobank — total ?; splits: train (-1), val (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve, measuring performance on the positive class across classification thresholds.

Input / output format

Input: Serialized tabular data formatted as JSON-like key-value pairs combined with a base prompt (e.g., 'Predict if a patient has the condition or not. bmi: {val}. age: {val}.'). For multimodal HeLM, non-text modalities (spirograms, tabular) are projected into the LLM's token embedding space via modality-specific encoders and concatenated with text tokens.

Output: The model generates a completion. The evaluation computes the log-likelihood of the completion being “yes}” or “no}” to derive a risk score.

Scoring recipe

# Per instance
ll_yes = model.log_likelihood(prompt + "yes}")
ll_no = model.log_likelihood(prompt + "no}")
risk_score = ll_yes  # or ll_yes - ll_no

# Aggregate across dataset
auroc = compute_auroc(risk_scores, true_labels)
auprc = compute_auprc(risk_scores, true_labels)

Common pitfalls

• Risk scores are derived from the log-likelihood of specific token completions ('yes}' or 'no}') rather than standard classification probabilities or logits.
• The evaluation is restricted to a European ancestry subset of UK Biobank, which may introduce population bias and limit cross-ancestry generalizability.
• Spirogram inputs are preprocessed into fixed-length 1D flow-volume curves, meaning the model does not evaluate raw time-series directly.

Evidence (verbatim from paper)

We then compute the log-likelihood of the sentence being completed with “yes}” or “no}”. This log-likelihood serves as a risk score and can be evaluated using metrics such as AUROC and AUPRC to assess discriminatory power.

Citation

@misc{belyaeva2023multimodal,
  title={Multimodal LLMs for health grounded in individual-specific data},
  author={Belyaeva et al. (2023)},
  year={2023},
  note={arXiv:2307.09018}
}

• arXiv: 2307.09018