Odd Eval

odd-eval

ODD: A Benchmark Dataset for the Natural Language Processing based Opioid Related Aberrant Behavior Detection — Kwon et al. (2023) (arXiv:2307.02591, 2023)

What this evaluates

This benchmark evaluates a model's ability to perform multi-label classification on clinical electronic health record (EHR) notes to detect nine categories of Opioid-Related Aberrant Behaviors (ORABs). It probes the model's capacity to identify both confirmed and suggested aberrant behaviors, as well as auxiliary opioid-related signals, under conditions of significant label imbalance.

Datasets

• ODD — total ?; splits: train (-1), test (-1); repo https://github.com/soon91jae/O_RAB_MIMIC

Metrics

• macro average AUPRC (primary) — range: [0, 1]
  • The unweighted mean of the Area Under the Precision-Recall Curve (AUPRC) computed independently for each of the 9 ORAB categories, then averaged across all classes.
• macro average F1 — range: [0, 1]
  • The unweighted mean of the F1 score computed independently for each of the 9 ORAB categories, then averaged across all classes.

Input / output format

Input: Clinical text excerpts from Electronic Health Record (EHR) notes.

Output: Multi-label binary classification vector indicating presence or absence across 9 Opioid-Related Aberrant Behavior (ORAB) categories.

Scoring recipe

def compute_macro_metrics(y_true, y_pred, n_classes=9):
    class_auprc = []
    class_f1 = []
    for c in range(n_classes):
        class_auprc.append(auprc(y_true[:, c], y_pred[:, c]))
        class_f1.append(f1_score(y_true[:, c], y_pred[:, c], average='binary'))
    return {
        'macro_avg_auprc': sum(class_auprc) / n_classes,
        'macro_avg_f1': sum(class_f1) / n_classes
    }

Common pitfalls

• Severe label imbalance causes models to perform poorly on rare classes (e.g., Suggested Aberrant Behaviors) while dominant classes achieve near-perfect scores.
• Macro-averaging masks class-level performance disparities; readers must inspect per-class metrics to understand model behavior on imbalanced data.
• The evaluation uses nested cross-validation (5 outer, 2 inner folds) for hyperparameter tuning, which differs from standard single train/test splits and affects variance reporting.

Evidence (verbatim from paper)

Models achieved a performance range of [77.91, 88.17] in macro average AUPRC and [70.40, 82.86] in macro average F1. Notably, the prompt-based fine-tuning models significantly outperformed the standard fine-tuning models in both the BioClinicalBERT and BioBERT frameworks, with an increase of 9.64 points and 9.65 points in macro AUPRC, respectively.

Citation

@misc{kwon2023odd,
  title={ODD: A Benchmark Dataset for the Natural Language Processing based Opioid Related Aberrant Behavior Detection},
  author={Kwon et al. (2023)},
  year={2023},
  note={arXiv:2307.02591}
}

• arXiv: 2307.02591