Mimic Iv Icd Eval

mimic-iv-icd-eval

Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification — Thanh-Tung Nguyen et al. (2023) (arXiv:2304.13998, 2023)

What this evaluates

Predicts ICD-9 and ICD-10 medical billing codes from clinical discharge notes under extreme multi-label classification settings. It probes a model's ability to handle long-tailed label distributions, high cardinality, and code hierarchy variations in electronic health records.

Datasets

• MIMIC-IV-ICD9 — total 209352; splits: train (188533), val (7110), test (13709); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing
• MIMIC-IV-ICD10 — total 122310; splits: train (110442), val (4017), test (7851); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing
• MIMIC-IV-ICD9-50 — total 189475; splits: train (170664), val (6406), test (12405); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing
• MIMIC-IV-ICD10-50 — total 115250; splits: train (104077), val (3805), test (7368); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing

Metrics

• Macro-F1 (primary) — range: [0, 1]
  • Standard macro-averaged F1 score computed over all unique ICD codes. Calculated as the unweighted mean of F1 scores for each label, where F1 = 2 * (Precision * Recall) / (Precision + Recall).
• Precision@K — range: [0, 1]
  • Fraction of predicted codes in the top-K ranked list that are correct. K is typically set to 5 or 10 for extreme multi-label settings.

Input / output format

Input: Clinical discharge note text (string). Optional patient metadata (age, gender, hospital stay duration) may be appended depending on the model variant.

Output: A ranked list or set of predicted ICD-9 or ICD-10 codes (strings) corresponding to the input note.

Scoring recipe

def compute_macro_f1(preds, golds):
    all_labels = set(golds) | set(preds)
    f1_scores = []
    for label in all_labels:
        tp = sum(1 for p in preds if label in p and label in golds)
        fp = sum(1 for p in preds if label in p and label not in golds)
        fn = sum(1 for p in preds if label not in p and label in golds)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores) if f1_scores else 0

Common pitfalls

• Splitting data by hospital admission (hadm_id) instead of patient (subject_id) causes patient-level data leakage across train/val/test sets.
• Ignoring the long-tailed distribution of ICD codes leads models to only predict frequent codes and fail on rare diagnoses.
• Inconsistent mapping of parent/child code hierarchies between ICD-9 and ICD-10 versions can cause evaluation mismatches if not standardized.

Evidence (verbatim from paper)

Evaluating existing methods from MIMIC-III in the MIMIC-IV context is advantageous for determining their performance in larger and more complex multilabel classification scenarios.

Citation

@misc{nguyen2023mimicivicd,
  title={Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification},
  author={Thanh-Tung Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2304.13998}
}

• arXiv: 2304.13998