Molecule3d Eval

molecule3d-eval

Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs — Zhao Xu et al. (2021) (arXiv:2110.01717, 2021)

What this evaluates

This benchmark evaluates the ability of graph neural networks to predict ground-state 3D molecular geometries directly from 2D molecular graphs, and subsequently assesses how well these predicted geometries improve downstream quantum property prediction (HOMO-LUMO gap).

Datasets

• Molecule3D — total 4000000; splits: val (-1), test (-1); repo https://github.com/divelab/MoleculeX

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: $\frac{1}{m}\sum_{i=1}^{m}|\hat{y}{i}-y{i}|$, where $\hat{y}{i}$ and $y{i}$ are predicted and ground-truth values (pairwise distances, coordinates, or HOMO-LUMO gaps). Lower values indicate better performance.
• RMSE — range: other
  • Root Mean Squared Error: $\sqrt{\frac{1}{m}\sum_{i=1}^{m}(\hat{y}{i}-y{i})^2}$. Sensitive to large errors in distance or coordinate prediction.
• Validity — range: percent
  • Percentage of predicted geometries that form a valid Euclidean Distance Matrix (EDM) and can be transformed into 3D coordinates.
• Validity3D — range: percent
  • Percentage of predicted geometries that are valid in 3D space (dimension ≤ 3) and satisfy triangle inequalities.

Input / output format

Input: Molecular graph with 9-dimensional node features (atomic number, chirality, hybridization) and 3-dimensional edge features (bond type, stereochemistry, conjugation).

Output: Predicted pairwise atomic distances (via element-wise max and linear transformation of node representations) or predicted 3D atomic coordinates (3D vectors). For property prediction: scalar HOMO-LUMO gap value.

Scoring recipe

def compute_mae(preds, gold):
    return np.mean(np.abs(preds - gold))

def compute_rmse(preds, gold):
    return np.sqrt(np.mean((preds - gold) ** 2))

def compute_validity(preds):
    # Check if predicted EDM is valid and transformable to 3D
    return np.mean(is_valid_3d_geometry(preds)) * 100

Common pitfalls

• Scaffold split is significantly harder than random split due to unseen molecular scaffolds, causing substantial performance drops.
• Predicting pairwise distances yields lower MAE/RMSE but extremely low Validity/Validity3D, while direct coordinate prediction guarantees 100% validity but higher distance errors.
• RDKit ETKDG baseline fails to generate geometries for a non-trivial number of molecules (1,311–4,434 depending on split), which must be accounted for when comparing success rates.

Evidence (verbatim from paper)

We evaluate the prediction performance by the mean absolute error (MAE) between the predicted properties and the ground-truth properties. Given a dataset of $m$ molecules whose HOMO-LUMO gaps are ${\hat{y}{i}}{i=1}^{m}$, and the predicted HOMO-LUMO gaps are ${y_{i}}{i=1}^{m}$, where $y{i},\hat{y}{i}\in\mathbb{R}$, the MAE is defined as: $\mbox{MAE}\left({\hat{y}{i}}{i=1}^{m},{y{i}}{i=1}^{m}\right)=\frac{1}{m}\sum{i=1}^{m}|\hat{y}{i}-y{i}|.$

Citation

@misc{xu2021molecule3d,
  title={Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs},
  author={Zhao Xu et al. (2021)},
  year={2021},
  note={arXiv:2110.01717}
}

• arXiv: 2110.01717