# Nrbdmf Drug Effect Prediction Eval

> Evaluates recommendation algorithms for predicting drug-target and drug-disease interactions. It specifically probes the model's ability to handle bidirectional drug effects (therapeutic vs. adverse) and rank candidate pairs accurately under cold-start cross-validation scenarios. Use when the user wants to benchmark on Drug-Protein benchmark dataset, Drug-Disease benchmark dataset, or asks about evaluating this task. Reports AUPR.

- Skill: `qhjqhj00/nrbdmf-drug-effect-prediction-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/nrbdmf-drug-effect-prediction-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/nrbdmf-drug-effect-prediction-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/nrbdmf-drug-effect-prediction-eval

---


# nrbdmf-drug-effect-prediction-eval

> NRBdMF: A recommendation algorithm for predicting drug effects considering directionality — Azuma et al. (2022) (arXiv:2208.04312, 2022)

## What this evaluates

Evaluates recommendation algorithms for predicting drug-target and drug-disease interactions. It specifically probes the model's ability to handle bidirectional drug effects (therapeutic vs. adverse) and rank candidate pairs accurately under cold-start cross-validation scenarios.

## Datasets

- **Drug-Protein benchmark dataset** — total ?; splits: CVS1 (-1), CVS2 (-1), CVS3 (-1); repo https://github.com/mizuno-group/NRBdMF
- **Drug-Disease benchmark dataset** — total ?; splits: CVS1 (-1), CVS2 (-1), CVS3 (-1); repo https://github.com/mizuno-group/NRBdMF

## Metrics

- `AUPR` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve, measuring the trade-off between precision and recall across all classification thresholds.
- `AUROC` — range: [0, 1]
  - Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate.
- `Enrichment Score (ES)` — range: other
  - Difference in enrichment values for side effects (+1) and indications (-1), calculated using statistics from gene set variation analysis (GSVA). Higher values indicate better separation of therapeutic vs. adverse effects.

## Input / output format

**Input**: Multilabel interaction matrix (values: +1 for side effects, -1 for indications, 0 for unknown) and precomputed similarity square matrices (kernels) for drugs and diseases/targets.

**Output**: Continuous prediction scores for each drug-disease/drug-protein pair, used to generate a ranked list of candidates.

## Scoring recipe

```python
def compute_aupr(y_true, y_pred):
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred)
    return auc(recalls, precisions)

def compute_auroc(y_true, y_pred):
    fprs, tprs, _ = roc_curve(y_true, y_pred)
    return auc(fprs, tprs)

def compute_es(y_true_pos, y_pred_pos, y_true_neg, y_pred_neg):
    enrich_pos = gsva_enrichment(y_true_pos, y_pred_pos)
    enrich_neg = gsva_enrichment(y_true_neg, y_pred_neg)
    return enrich_pos - enrich_neg
```

## Common pitfalls

- CVS2 and CVS3 settings mask entire drugs or diseases, simulating realistic cold-start scenarios that drastically lower performance compared to CVS1.
- Standard binary metrics like AUPR/AUROC do not capture the bidirectional nature of drug effects; the Enrichment Score (ES) is required to evaluate the separation of therapeutic vs. adverse predictions.
- Grid search for hyperparameters is performed on training/validation splits, but final evaluation is strictly on held-out test folds to prevent data leakage.

## Evidence (verbatim from paper)

> The performance of the seven representative algorithms was evaluated using two types of benchmark datasets by 10-fold CV performed five times in the three scenarios. The area under precision-recall curve (AUPR) of each method in predicting drug-target proteins is shown in Fig. 2b... The prediction performance was evaluated for the remaining test datasets using the optimized version of each model... The difference between the enrichment of side effects and the enrichment of indications is shown as an ES in Fig. 4b. In CVS3, the mean ES of NRBdMF (0.588 ± 0.101) outperformed that of NRLMF (0.191 ± 0.106).

## Citation

```bibtex
@misc{azuma2022nrbdmf,
  title={NRBdMF: A recommendation algorithm for predicting drug effects considering directionality},
  author={Azuma et al. (2022)},
  year={2022},
  note={arXiv:2208.04312}
}
```

- arXiv: 2208.04312