HolobiomicsLab auc-metric-calculation
Summary
Compute the Area Under the Curve (AUC) metric from ROC curve data to quantify binary classification model performance. AUC summarizes classification discrimination ability as a single scalar between 0 and 1, suitable for publication-quality reporting alongside ROC visualizations.
When to use
When you have computed true positive rate (TPR) and false positive rate (FPR) values across classification probability thresholds from predicted probabilities and true labels, and need to generate a single aggregate performance metric for model comparison or reporting in omics classification tasks.
When NOT to use
- Input data contains only a single class or lacks binary labels — AUC is undefined for single-class or multi-class problems without one-vs-rest conversion.
- Predicted probabilities or true labels are missing or have mismatched dimensions — AUC calculation requires complete paired observations.
- The classification task is multi-class without explicit binary reduction — standard AUC applies to binary outcomes.
Inputs
- CSV file with predicted probabilities and binary true labels (e.g., roc_example.csv)
- Computed pairs of true positive rate and false positive rate across thresholds
Outputs
- AUC scalar metric (numeric value between 0 and 1)
- ROC curve plot with AUC annotated and rendered as PNG or PDF
How to apply
After loading predicted probabilities and binary true labels from a CSV file (e.g., roc_example.csv), calculate TPR and FPR pairs across all classification thresholds using statistical libraries (R or Python). The AUC is then computed as the area under the ROC curve using numerical integration (trapezoidal rule or equivalent). The resulting AUC value (ranging from 0 to 1, with 0.5 indicating random classifier performance) should be displayed on the ROC curve plot alongside the visualization and saved as part of the publication-quality figure output (PNG or PDF).
Related tools
- R Shiny (Interactive dashboard framework for executing ROC curve analysis and AUC calculation, rendering results with annotations) — github.com/databio2022/GraphBio
Evaluation signals
- AUC value is a scalar numeric between 0 and 1 (exclusive of invalid boundary cases like NaN or out-of-range values)
- AUC is correctly displayed on the ROC curve plot as a text annotation (e.g., 'AUC = 0.87')
- Plot is rendered as publication-quality PNG or PDF with clear axes, legend, and AUC label visible
- AUC value is consistent with manual trapezoidal integration of the TPR/FPR curve pairs
- For a random classifier (diagonal line from (0,0) to (1,1)), AUC should be approximately 0.5
Limitations
- AUC does not indicate optimal operating threshold; threshold selection requires domain knowledge or additional metrics (sensitivity/specificity trade-off).
- AUC can be misleading for highly imbalanced datasets where class prevalence is extreme; precision-recall curves or threshold-adjusted metrics may be more informative.
- The method assumes predicted probabilities are well-calibrated; poorly calibrated probability estimates may produce misleading AUC values.
Evidence
- [other] task_id=task_004 workflow definition: "Compute the Area Under the Curve (AUC) metric from the ROC curve."
- [readme] GraphBio ROC curve capability: "roc_example.csv for ROC curve"
- [other] ROC curve analysis workflow from task card: "Calculate the ROC curve by computing true positive rate and false positive rate across classification thresholds using R or Python statistical libraries."
- [other] Output format specification: "Render the ROC curve as a publication-quality figure with AUC displayed on the plot and save as a PNG or PDF file."