Marcel Eval

marcel-eval

Learning Over Molecular Conformer Ensembles: Datasets and Benchmarks — Zhu et al. (2023) (arXiv:2310.00115, 2023)

What this evaluates

Evaluates molecular property prediction using explicit conformer ensembles versus single-conformer or 1D/2D baselines, probing how 3D structural flexibility and ensemble encoding strategies impact regression accuracy.

Datasets

• MARCEL — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/SXKDZ/MARCEL

Metrics

• Mean Absolute Error (MAE) (primary) — range: other
  • MAE = (1/N) Σ |y_pred - y_true| over all test instances. Lower values indicate better performance.

Input / output format

Input: Molecular structures represented as 1D sequences, 2D graphs, or 3D conformer ensembles (capped at 20 conformers per molecule).

Output: A single scalar value predicting the target molecular property.

Scoring recipe

def compute_mae(preds, gold):
    errors = [abs(p - g) for p, g in zip(preds, gold)]
    return sum(errors) / len(errors)

Common pitfalls

• Uniform random sampling of conformers during training ignores Boltzmann weights, which can hurt performance on tasks sensitive to energy distributions.
• 3D models do not consistently outperform 1D/2D baselines on small datasets or properties insensitive to conformational changes.
• Capping encoded conformers to 20 per molecule is a stability trick that may alter learning dynamics and should be reported.

Evidence (verbatim from paper)

Each dataset is partitioned randomly into three subsets: 70% for training, 10% for validation, and 20% for test. ... The metric used is the Mean Absolute Error (MAE, ↓).

Citation

@misc{zhu2023marcel,
  title={Learning Over Molecular Conformer Ensembles: Datasets and Benchmarks},
  author={Zhu et al. (2023)},
  year={2023},
  note={arXiv:2310.00115}
}

• arXiv: 2310.00115