Protein Graph Embedding Eval

protein-graph-embedding-eval

Neural Embeddings for Protein Graphs — Ceccarelli et al. (2023) (arXiv:2306.04667, 2023)

What this evaluates

Evaluates the ability of graph neural networks combined with language models to learn structural and sequence representations of proteins. It probes how well the learned embeddings preserve structural similarity via TM-score prediction and generalize to downstream classification tasks across different protein families and out-of-distribution datasets.

Datasets

• Kinase dataset — total ?; splits: train (-1), test (-1)
• SCOPe dataset — total 13265; splits: test (13265)

Metrics

• MSE (primary) — range: [0, inf)
  • Mean Squared Error between predicted and true TM-scores or structural distances. Lower is better.
• Pearson correlation — range: [-1, 1]
  • Pearson correlation coefficient between model predictions and true TM-scores.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified kinase family labels out of total samples.
• F1-score — range: [0, 1]
  • Average F1-score across 5-fold cross-validation for protein family classification.

Input / output format

Input: Protein graphs constructed from amino acid sequences (using LSTM or BERT extracted features) and 3D structural topology (node connections).

Output: Graph-level embeddings of size 256, or predicted TM-scores, or predicted protein family labels.

Scoring recipe

def compute_metrics(preds, gold, task='tm_score'):
    if task == 'tm_score':
        mse = np.mean((preds - gold) ** 2)
        pearson = np.corrcoef(preds, gold)[0, 1]
        return {'MSE': mse, 'Pearson': pearson}
    else:
        acc = np.mean(preds == gold)
        f1 = f1_score(gold, preds, average='macro')
        return {'Accuracy': acc, 'F1': f1}

Common pitfalls

• Performance heavily depends on the choice of distance function (Cosine, Euclidean, Manhattan, Square) used in the embedding space, with Euclidean and Square consistently yielding lower MSE and higher accuracy.
• Node feature extraction method drastically impacts results; LLM-extracted features (LSTM/BERT) significantly outperform handcrafted features (One-hot, BLOSUM, Physicochemical), making direct comparison with older baselines unfair without feature alignment.
• The OOD evaluation trains exclusively on kinase proteins and tests on SCOPe, so reported F1-scores measure transfer learning capability rather than direct supervised performance on the target domain.

Evidence (verbatim from paper)

We evaluate the proposed framework using Graph Convolutional Networks (GCNs), Graph Attention Networks (GATs), and GraphSAGE... All the models were implemented with two graph layers in PyTorch geometric to learn protein embeddings of size 256. For the generation of the embeddings, we used 80% of the kinase proteins for training and the remaining 20% for testing. Table 2 shows the MSE values for the graph encoders... The accuracy of classification, computed as the average result of 5-fold cross-validation... Results of this evaluation, measured as average F1-score across 5 folds for each distance function, are shown in Table 4

Citation

@misc{ceccarelli2023neural,
  title={Neural Embeddings for Protein Graphs},
  author={Ceccarelli et al. (2023)},
  year={2023},
  note={arXiv:2306.04667}
}

• arXiv: 2306.04667