Drugbank Hetionet Eval

drugbank-hetionet-eval

Fast Dual-Regularized Autoencoder for Sparse Biological Data — Poleksic (2024) (arXiv:2401.16664, 2024)

What this evaluates

Evaluates the ability of matrix completion algorithms to predict missing biological interactions (drug-target or compound-disease) using sparse association matrices and side information.

Datasets

• DrugBank — total 9881; splits: 5-fold CV (3 rounds) (-1); repo www.cs.uni.edu/~poleksic/drugbank_files.tar.gz
• Hetionet (Drug Repurposing) — total ?; splits: 5-fold CV (3 rounds) (-1); repo www.cs.uni.edu/~poleksic/hetionet_files.tar.gz

Metrics

• AUPR (primary) — range: [0, 1]
  • Area Under the Precision-Recall curve. Computed by integrating precision over recall across all classification thresholds.
• NDCG@100 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 100. Measures ranking quality by discounting relevance logarithmically with position and normalizing by the ideal DCG.
• PREC@50 — range: [0, 1]
  • Precision at rank 50. The fraction of true positive interactions among the top 50 predicted items.
• PREC@100 — range: [0, 1]
  • Precision at rank 100. The fraction of true positive interactions among the top 100 predicted items.

Input / output format

Input: Sparse binary interaction matrix (e.g., drug-target or compound-disease) and side-information similarity matrices (e.g., drug-drug, target-target, or gene-profile-based similarities).

Output: Predicted interaction probabilities or scores for all missing entries in the interaction matrix.

Scoring recipe

def compute_metrics(y_true, y_pred, k=50):
    indices = np.argsort(y_pred)[::-1]
    y_true_sorted = y_true[indices]
    precisions = np.cumsum(y_true_sorted) / np.arange(1, len(y_true_sorted) + 1)
    recalls = np.cumsum(y_true_sorted) / np.sum(y_true)
    auprc = np.trapz(precisions, recalls)
    dcg = np.sum(y_true_sorted[:100] / np.log2(np.arange(2, 102)))
    idcg = np.sum(np.ones(min(100, np.sum(y_true))) / np.log2(np.arange(2, min(100, np.sum(y_true)) + 2)))
    ndcg = dcg / idcg if idcg > 0 else 0
    prec_k = np.mean(y_true_sorted[:k])
    return auprc, ndcg, prec_k

Common pitfalls

• Random 5-fold CV splits do not account for temporal or biological constraints in drug discovery, potentially inflating performance.
• Hyperparameter tuning is extensive and significantly impacts results; baselines must be thoroughly optimized for fair comparison.
• Baseline implementations (e.g., COSINE/NRLMF equivalence) may differ from standard open-source versions, requiring careful reproduction.

Evidence (verbatim from paper)

Our benchmarking procedure uses three rounds of the classical 5-fold cross-validation (CV). In each CV round, the input drug-target association matrix $X$ is randomly split into 5 groups. Each group is used once as test data, while the remaining four groups represent training data. The final classification scores (AUPR, NDCG100, PREC@50, and PREC@100) are computed by averaging the classification scores obtained across different CV rounds.

Citation

@misc{poleksic2024fast,
  title={Fast Dual-Regularized Autoencoder for Sparse Biological Data},
  author={Poleksic (2024)},
  year={2024},
  note={arXiv:2401.16664}
}

• arXiv: 2401.16664