Glycanml Eval

glycanml-eval

GlycanML: A Multi-Task and Multi-Structure Benchmark for Glycan Machine Learning — Xu et al. (2024) (arXiv:2405.16206, 2024)

What this evaluates

Evaluates machine learning models on glycan analysis tasks, including taxonomic classification, immunogenicity prediction, glycosylation type prediction, and protein-glycan binding affinity estimation. It probes the ability of sequence-based and graph-based encoders to capture multi-relational glycan structures and benefit from multi-task learning.

Datasets

• GlycanML — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/GlycanML/GlycanML

Metrics

• Macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across all classes for each taxonomy sub-task and the glycosylation type task.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve for binary immunogenicity prediction.
• Spearman’s ρ — range: [-1, 1]
  • Spearman rank correlation coefficient between predicted and true protein-glycan binding affinity values.
• Weighted Mean Rank — range: other
  • Comprehensive metric averaging the rank of each model across all 11 tasks, where each of the 8 taxonomy tasks weighs 1/8 and the other 3 tasks weigh 1.

Input / output format

Input: Glycan structures represented as sequences or graphs (homogeneous/heterogeneous). For interaction prediction, protein sequences are encoded via ESM-1b and concatenated with glycan representations.

Output: Predicted class labels for taxonomy, immunogenicity, and glycosylation tasks; predicted continuous binding affinity score for interaction prediction.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None, y_true_cont=None, y_pred_cont=None):
    macro_f1 = f1_score(y_true, y_pred, average='macro')
    auprc = average_precision_score(y_true, y_prob)
    spearman_rho = spearmanr(y_true_cont, y_pred_cont).correlation
    ranks = [rankdata(-scores[task]) for task in tasks]
    wmr = (sum(ranks[:8]) / 8) + sum(ranks[8:])
    return macro_f1, auprc, spearman_rho, wmr

Common pitfalls

• Using homogeneous GNNs ignores glycosidic bond types, leading to suboptimal performance compared to heterogeneous/multi-relational GNNs.
• Applying small-molecule encoders (e.g., Graphormer) directly to glycans fails due to scale mismatch, as glycans are macromolecules with hundreds of atoms.
• Assuming multi-task learning (MTL) universally improves performance; the paper shows most MTL methods degrade results due to difficulty in balancing correlated taxonomy tasks.

Evidence (verbatim from paper)

We measure the comprehensive performance of a model with its weighted mean rank over all tasks, where each taxonomy prediction task weighs 1/8 and each of the other three tasks weighs 1, so as to balance between different types of tasks.

Citation

@misc{xu2024glycanml,
  title={GlycanML: A Multi-Task and Multi-Structure Benchmark for Glycan Machine Learning},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2405.16206}
}

• arXiv: 2405.16206