Reactembed Eval

reactembed-eval

ReactEmbed: A Cross-Domain Framework for Protein-Molecule Representation Learning via Biochemical Reaction Networks — Sicherman et al. (2025) (arXiv:2501.18278, 2025)

What this evaluates

Evaluates a cross-domain representation learning framework for protein-molecule interactions by measuring prediction accuracy on regression and classification tasks across diverse biochemical benchmarks.

Datasets

• FreeSolv — total ?; splits: test (-1)
• CEP — total ?; splits: test (-1)
• BetaLactamase — total ?; splits: test (-1)
• Stability — total ?; splits: test (-1)
• BindingDB — total ?; splits: test (-1)
• PPIAffinity — total ?; splits: test (-1)
• BBBP — total ?; splits: test (-1)
• GO-CC — total ?; splits: test (-1)
• DrugBank — total ?; splits: test (-1)
• HumanPPI — total ?; splits: test (-1)
• YeastPPI — total ?; splits: test (-1)

Metrics

• Root Mean Square Error (RMSE) (primary) — range: other
  • The square root of the average of squared differences between predicted and actual values. Lower values indicate better performance.
• Area Under the Curve (AUC) — range: [0, 1]
  • The area under the Receiver Operating Characteristic (ROC) curve, measuring the model's ability to distinguish between classes across all classification thresholds. Higher values indicate better performance.

Input / output format

Input: Protein sequences/structures and molecular graphs/SMILES paired via biochemical reaction networks.

Output: Continuous values for regression tasks or class probabilities/labels for classification tasks.

Scoring recipe

def rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_true - y_pred) ** 2))

def auc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)

Common pitfalls

• Positive Δ% values in the tables indicate improvement (reduction in RMSE for regression), which is counterintuitive.
• Evaluations compare against multiple baseline combinations (ESM3/GearNet/ProtBert × MolCLR/MolFormer) rather than a single fixed baseline, complicating direct cross-task comparisons.
• Statistical significance is reported at p < 0.05, but the specific statistical test used (e.g., paired t-test) is not detailed in the results section.

Evidence (verbatim from paper)

Table 1: Evaluation of ReactEmbed against pre-trained baselines for regression tasks, using Root Mean Square Error (RMSE) as the evaluation metric (lower is better). The Δ% values show the absolute change and relative percentage improvement compared to the pre-trained baseline, where positive values indicate improvement (reduction in RMSE).

Citation

@misc{sicherman2025reactembed,
  title={ReactEmbed: A Cross-Domain Framework for Protein-Molecule Representation Learning via Biochemical Reaction Networks},
  author={Sicherman et al. (2025)},
  year={2025},
  note={arXiv:2501.18278}
}

• arXiv: 2501.18278