Nrbdmf Drug Effect Prediction Eval

nrbdmf-drug-effect-prediction-eval

NRBdMF: A recommendation algorithm for predicting drug effects considering directionality — Azuma et al. (2022) (arXiv:2208.04312, 2022)

What this evaluates

Evaluates recommendation algorithms for predicting drug-target and drug-disease interactions. It specifically probes the model's ability to handle bidirectional drug effects (therapeutic vs. adverse) and rank candidate pairs accurately under cold-start cross-validation scenarios.

Datasets

• Drug-Protein benchmark dataset — total ?; splits: CVS1 (-1), CVS2 (-1), CVS3 (-1); repo https://github.com/mizuno-group/NRBdMF
• Drug-Disease benchmark dataset — total ?; splits: CVS1 (-1), CVS2 (-1), CVS3 (-1); repo https://github.com/mizuno-group/NRBdMF

Metrics

• AUPR (primary) — range: [0, 1]
  • Area under the precision-recall curve, measuring the trade-off between precision and recall across all classification thresholds.
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate.
• Enrichment Score (ES) — range: other
  • Difference in enrichment values for side effects (+1) and indications (-1), calculated using statistics from gene set variation analysis (GSVA). Higher values indicate better separation of therapeutic vs. adverse effects.

Input / output format

Input: Multilabel interaction matrix (values: +1 for side effects, -1 for indications, 0 for unknown) and precomputed similarity square matrices (kernels) for drugs and diseases/targets.

Output: Continuous prediction scores for each drug-disease/drug-protein pair, used to generate a ranked list of candidates.

Scoring recipe

def compute_aupr(y_true, y_pred):
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred)
    return auc(recalls, precisions)

def compute_auroc(y_true, y_pred):
    fprs, tprs, _ = roc_curve(y_true, y_pred)
    return auc(fprs, tprs)

def compute_es(y_true_pos, y_pred_pos, y_true_neg, y_pred_neg):
    enrich_pos = gsva_enrichment(y_true_pos, y_pred_pos)
    enrich_neg = gsva_enrichment(y_true_neg, y_pred_neg)
    return enrich_pos - enrich_neg

Common pitfalls

• CVS2 and CVS3 settings mask entire drugs or diseases, simulating realistic cold-start scenarios that drastically lower performance compared to CVS1.
• Standard binary metrics like AUPR/AUROC do not capture the bidirectional nature of drug effects; the Enrichment Score (ES) is required to evaluate the separation of therapeutic vs. adverse predictions.
• Grid search for hyperparameters is performed on training/validation splits, but final evaluation is strictly on held-out test folds to prevent data leakage.

Evidence (verbatim from paper)

The performance of the seven representative algorithms was evaluated using two types of benchmark datasets by 10-fold CV performed five times in the three scenarios. The area under precision-recall curve (AUPR) of each method in predicting drug-target proteins is shown in Fig. 2b... The prediction performance was evaluated for the remaining test datasets using the optimized version of each model... The difference between the enrichment of side effects and the enrichment of indications is shown as an ES in Fig. 4b. In CVS3, the mean ES of NRBdMF (0.588 ± 0.101) outperformed that of NRLMF (0.191 ± 0.106).

Citation

@misc{azuma2022nrbdmf,
  title={NRBdMF: A recommendation algorithm for predicting drug effects considering directionality},
  author={Azuma et al. (2022)},
  year={2022},
  note={arXiv:2208.04312}
}

• arXiv: 2208.04312