Hip Dislocation Detection Eval

hip-dislocation-detection-eval

Natural Language Processing with Deep Learning for Medical Adverse Event Detection from Free-Text Medical Narratives: A Case Study of Detecting Total Hip Replacement Dislocation — Borjali et al. (2020) (arXiv:2004.08333, 2020)

What this evaluates

This benchmark probes a model's ability to detect adverse events (total hip replacement dislocation) from unstructured, free-text clinical narratives. It evaluates whether NLP models can correctly classify medical notes into dislocation status categories, handling complex negation, long-range dependencies, and multi-site anatomical references.

Datasets

• Radiology Notes — total ?; splits: test (301)
• Telephone Notes — total ?; splits: test (79)

Metrics

• Kappa (primary) — range: [-1, 1]
  • Cohen's/Fleiss' Kappa coefficient measuring agreement between predicted and gold labels, adjusted for chance. Calculated per model on the held-out test set.

Input / output format

Input: Free-text clinical narratives (radiology reports or conversational follow-up telephone notes) describing patient status post total hip replacement.

Output: Categorical label: for radiology notes, one of ['no dislocation', 'current dislocation', 'evidence of previous dislocation']; for telephone notes, one of ['no dislocation', 'evidence of previous dislocation'].

Scoring recipe

def compute_kappa(predictions, gold):
    # predictions and gold are lists of class labels
    # Use standard kappa implementation (e.g., sklearn.metrics.cohen_kappa_score)
    return kappa_score(gold, predictions)

# Per-class precision and recall are also reported:
# precision_c = true_positives_c / (true_positives_c + false_positives_c)
# recall_c = true_positives_c / (true_positives_c + false_negatives_c)

Common pitfalls

• Models struggle with long-range dependencies and negation, often misclassifying notes where 'no' is far from 'dislocation' (e.g., 'no fracture or dislocation').
• Multi-site anatomical mentions cause misclassification; notes discussing dislocation at other sites (knee, rib) while the hip is actually dislocated are often labeled 'no dislocation'.
• Relying on structured ICD/CPT codes significantly underreports adverse events compared to free-text analysis, leading to false negatives in baseline comparisons.

Evidence (verbatim from paper)

Both DL-NLP models (proposed CNN and LSTM) outperformed all ML-NLP models and achieved the highest and second highest Kappa score. The CNN model achieved the highest overall results for all three classes.

Citation

@misc{borjali2020nlp,
  title={Natural Language Processing with Deep Learning for Medical Adverse Event Detection from Free-Text Medical Narratives: A Case Study of Detecting Total Hip Replacement Dislocation},
  author={Borjali et al. (2020)},
  year={2020},
  note={arXiv:2004.08333}
}

• arXiv: 2004.08333