Fs Mol Eval

fs-mol-eval

Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Prediction — Chen et al. (2022) (arXiv:2205.02708, 2022)

What this evaluates

Few-shot molecular property prediction and regression on a large-scale benchmark with thousands of tasks. Probes generalization across diverse protein targets and varying support set sizes.

Datasets

• FS-Mol — total 233786; splits: train (4938), val (40), test (157)

Metrics

• ΔAUPRC (primary) — range: other
  • Change in area under the precision-recall curve between the model and a baseline. Used for binary classification tasks.
• R²_os — range: other
  • Predictive or out-of-sample coefficient of determination. Used for regression tasks.

Input / output format

Input: Molecular compounds associated with specific protein targets. Support sets of varying sizes (16, 32, 64, 128, 256) for few-shot learning.

Output: Predicted activity values (binary or continuous IC50/EC50).

Scoring recipe

def score_fs_mol(predictions, labels, task_type):
    if task_type == 'classification':
        auprc = average_precision_score(labels, predictions)
        baseline_auprc = get_baseline_auprc()
        return auprc - baseline_auprc
    else:
        return r2_score(labels, predictions)
# Average over 10 stratified splits per task, then across tasks

Common pitfalls

• Support sets are generally unbalanced for classification, reflecting real-world drug discovery distributions.
• Metrics are reported as changes (Δ) from a baseline for classification, not raw AUPRC.

Evidence (verbatim from paper)

The task-level metrics for binary classification and regression are ΔAUPRC (change in area under the precision-recall curve) and R²_os (predictive/out-of-sample coefficient of determination), respectively. ... averaged performance over ten different stratified support/query random splits of every test task is reported for each compared method.

Citation

@misc{chen2022metalearning,
  title={Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Prediction},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2205.02708}
}

• arXiv: 2205.02708