Ici Response Prediction Eval

ici-response-prediction-eval

Transcriptomic Models for Immunotherapy Response Prediction Show Limited Cross-cohort Generalisability — Liang et al. (2026) (arXiv:2604.05478, 2026)

What this evaluates

Evaluates the cross-cohort generalisability of transcriptomic models (bulk and single-cell RNA-seq) for predicting immune checkpoint inhibitor (ICI) response in cancer patients. Probes robustness to cohort-specific transcriptomic context, tumour type, immune composition, and class imbalance.

Datasets

• Cho et al. — total ?; splits: test (-1)
• Ribas et al. — total ?; splits: test (-1)
• Poddubskaya et al. — total ?; splits: test (-1)
• Gondal et al. — total ?; splits: test (-1)
• Franken et al. — total ?; splits: test (-1)
• Luoma et al. — total ?; splits: test (-1)
• Reinstein et al. — total ?; splits: test (-1)

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of total instances.
• macro F1 score (primary) — range: [0, 1]
  • Unweighted mean of F1 scores calculated per class. Computed as 2 * (precision * recall) / (precision + recall) for each class, then averaged. Chosen as primary metric due to substantial class imbalance across cohorts.
• AUC — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the model's ability to discriminate between positive and negative classes across all classification thresholds.

Input / output format

Input: Transcriptomic expression data (bulk RNA-seq or scRNA-seq) from patient tumour samples, typically accompanied by clinical labels indicating immune checkpoint inhibitor (ICI) response status (responder vs. non-responder) and sometimes drug-specific annotations (PD-1/PD-L1).

Output: Binary classification prediction (responder vs. non-responder) or predicted probability of response.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob):
    acc = np.mean(y_true == y_pred)
    f1 = f1_score(y_true, y_pred, average='macro')
    auc = roc_auc_score(y_true, y_prob)
    return {'accuracy': acc, 'macro_f1': f1, 'AUC': auc}

Common pitfalls

• High AUC does not guarantee good classification performance under class imbalance; large discrepancies between AUC and macro F1 indicate poor minority-class prediction.
• Models may produce implausible perfect metrics (e.g., accuracy=1.00) on small, imbalanced cohorts due to overfitting or LOOCV-based retraining, inflating performance estimates.
• Cross-cohort generalisation is highly sensitive to tumour type, immune context, and technical heterogeneity, making direct performance comparisons across datasets misleading without accounting for cohort composition.

Evidence (verbatim from paper)

We evaluated all nine models on six independent, previously unseen cohorts and assessed predictive performance using accuracy, macro F1 score, and area under the receiver operating characteristic curve (AUC). ... Across all datasets, class imbalance was substantial, with imbalance ratios ranging from 0.25 to 0.94, underscoring the importance of macro F1 as a primary performance metric.

Citation

@misc{liang2026transcriptomic,
  title={Transcriptomic Models for Immunotherapy Response Prediction Show Limited Cross-cohort Generalisability},
  author={Liang et al. (2026)},
  year={2026},
  note={arXiv:2604.05478}
}

• arXiv: 2604.05478