Cohortgpt Medical Classification Eval

cohortgpt-medical-classification-eval

CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study — Guan et al. (2023) (arXiv:2307.11346, 2023)

What this evaluates

Evaluates large language models and fine-tuned baselines on multi-label medical text classification for clinical cohort recruitment. It probes the model's ability to extract and predict disease labels from unstructured radiology reports using few-shot prompting and knowledge graph augmentation.

Datasets

• IU-RR — total 3955; splits: train (-1), test (-1)
• MIMIC-CXR — total 227835; splits: train (-1), test (1808)

Metrics

• Exact Match Ratio (MR) — range: [0, 1]
  • The portion of complete correct predictions, averaged across all instances.
• Precision (P) — range: [0, 1]
  • The proportion of predicted correct labels to the total number of actual labels, averaged over all instances.
• Recall (R) — range: [0, 1]
  • The proportion of predicted correct labels to the total number of predicted labels.
• F1-Score (F) (primary) — range: [0, 1]
  • The harmonic mean of precision and recall.
• Hamming Loss (HL) — range: [0, 1]
  • Evaluates the average difference between predictions and ground truth.

Input / output format

Input: Concatenated 'findings' and 'impression' text from radiology reports, augmented with knowledge graph information and few-shot chain-of-thought examples in the prompt.

Output: A set of predicted disease labels (multi-label classification) corresponding to the 20 target diseases.

Scoring recipe

def compute_metrics(preds, golds):
    # preds, golds: list of sets of labels per instance
    exact_match = sum(1 for p, g in zip(preds, golds) if p == g) / len(preds)
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        tp += len(p & g)
        fp += len(p - g)
        fn += len(g - p)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    hl = (len(preds) * len(golds[0]) - tp) / (len(preds) * len(golds[0]))
    return exact_match, precision, recall, f1, hl

Common pitfalls

• MIMIC-CXR ground truth relies on CheXpert pseudo-labels rather than manual annotation, which may introduce label noise.
• Performance is highly sensitive to prompt construction (KG embedding strategy and CoT sample selection), not just the base model's inherent capability.
• The paper reports metrics averaged over instances but does not specify macro vs. micro averaging, which can yield different values for imbalanced multi-label tasks.

Evidence (verbatim from paper)

We adopt five popular metrics for the multi-label classification task as in [46]: Exact Match Ratio (MR), Precision (P), Recall (R), F1-Score (F), and Hamming Loss (HL). Exact Match Ratio is the portion of complete correct predictions, averaged across all instances, Precision is the proportion of predicted correct labels to the total number of actual labels, averaged over all instances, Recall is the proportion of predicted correct labels to the total number of predicted labels, F1-Score is the harmonic mean of precision and recall, Hamming Loss evaluates the average difference between predictions and ground truth.

Citation

@misc{guan2023cohortgpt,
  title={CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study},
  author={Guan et al. (2023)},
  year={2023},
  note={arXiv:2307.11346}
}

• arXiv: 2307.11346