Foresight2 Eval

foresight2-eval

Large Language Models for Medical Forecasting -- Foresight 2 — Kraljevic et al. (2024) (arXiv:2412.10848, 2024)

What this evaluates

Evaluates a fine-tuned LLM's ability to predict future biomedical concepts and clinical disorders from patient clinical timelines. It measures concept prediction accuracy via precision and recall across different temporal windows and candidate counts, and assesses clinical risk forecasting by checking how many of the top-5 predicted disorders match the ground truth for the next month.

Datasets

• MIMIC-III — total ?; splits: test (535)

Metrics

• Precision (primary) — range: [0, 1]
  • Ratio of correctly predicted concepts to the total number of predicted concepts. Evaluated as micro-average over all concept types for 'All' rows, and separately for New and Recurring concepts.
• Recall — range: [0, 1]
  • Ratio of correctly predicted concepts to the total number of actual ground-truth concepts. Evaluated as micro-average over all concept types for 'All' rows, and separately for New and Recurring concepts.
• At least N correct — range: percent
  • Percentage of patients where the number of correct predictions among the top-5 output disorders matches the ground truth disorders for the next month is greater than or equal to N (N=1, 2, 3).

Input / output format

Input: Structured patient timelines containing biomedical concepts and their temporal context (days relative to prediction point).

Output: Predicted next biomedical concept(s) or a ranked list of top 5 disorders for the next month.

Scoring recipe

def score_concept_predictions(predictions, golds):
    tp = sum(1 for p, g in zip(predictions, golds) if p == g)
    precision = tp / len(predictions) if predictions else 0
    recall = tp / len(golds) if golds else 0
    return precision, recall

def score_risk_forecasting(predictions_list, golds_list, n):
    correct_counts = []
    for preds, golds in zip(predictions_list, golds_list):
        matches = sum(1 for p in preds[:5] if p in golds)
        correct_counts.append(matches)
    at_least_n = sum(1 for c in correct_counts if c >= n) / len(golds_list) * 100
    return at_least_n

Common pitfalls

• Temporal window size (T-days) and number of candidates (@) drastically change scores; results are not directly comparable across different T/@ settings.
• Support sizes vary across models due to refusal to predict or sequence length limits, making direct percentage comparisons potentially misleading.
• Micro-averaging is used for 'All' concept types, which can mask performance differences on rare or specific concept categories.

Evidence (verbatim from paper)

Table 1: Results for the next concept prediction task. The ’All’ rows are calculated using the micro average over all concept types. ... Precision | Recall ... GPT-4-turbo, BioMistral, MedAlpaca, MEDITRON and FS2 were tasked with predicting the top 5 disorders a patient is at risk of in the next month. The dataset consisted of 535 patients from the test set prepared for the risk prediction task. As seen in Table [2], out of the 5 predictions on the dataset of 535 patients using FS2-Mistral, in 90% of patients, at least one prediction was correct.

Citation

@misc{kraljevic2024foresight2,
  title={Large Language Models for Medical Forecasting -- Foresight 2},
  author={Kraljevic et al. (2024)},
  year={2024},
  note={arXiv:2412.10848}
}

• arXiv: 2412.10848