Moleculenet F1 Eval

moleculenet-f1-eval

SmilesT5: Domain-specific pretraining for molecular language models — Philip Spence, Brooks Paige, Anne Osbourn (2025) (arXiv:2507.22514, 2025)

What this evaluates

Evaluates molecular property prediction by fine-tuning SMILES-based language models on standard chemical classification benchmarks. Probes the model's ability to learn structural chemistry from text representations and transfer that knowledge to downstream tasks.

Datasets

• MoleculeNet classification benchmarks — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. Calculated per benchmark across ten independent scaffold splits and averaged.
• BLEU score — range: [0, 1]
  • N-gram based metric for evaluating the similarity between predicted and target text sequences during pretraining.

Input / output format

Input: SMILES string representing a molecule.

Output: Classification label(s) corresponding to the molecular property benchmark.

Scoring recipe

def evaluate(predictions, gold, n_splits=10):
    f1_scores = []
    for i in range(n_splits):
        preds_i, gold_i = get_split_data(predictions, gold, i)
        f1 = f1_score(gold_i, preds_i, average='weighted')
        f1_scores.append(f1)
    mean_f1 = np.mean(f1_scores)
    return mean_f1, f1_scores

# Statistical comparison across models:
# p_value = mannwhitneyu(f1_scores_model_a, f1_scores_model_b, alternative='greater')

Common pitfalls

• Using random data splits instead of scaffold splitting causes data leakage due to molecular structural similarity.
• Relying on ROC AUC can overestimate model performance on test data compared to F1 score.
• Assuming larger pretraining datasets (100M vs 1M molecules) linearly improve downstream performance; results showed mixed or negligible gains.

Evidence (verbatim from paper)

Each benchmark dataset was split into training, validating, and testing datasets (with a ratio of 80/10/10) ten times using the scaffold splitting method and ten different random seeds. The F1 score, which is the harmonic mean between the precision and recall, was calculated using the predictions and ground truth of the test datasets. Typically the area under the curve of the receiver operator characteristic (ROC AUC) is used to compare molecular property prediction models, but some have found this metric to overestimate the model's performance against the test data [27].

Citation

@misc{spence2025smilest5,
  title={SmilesT5: Domain-specific pretraining for molecular language models},
  author={Philip Spence, Brooks Paige, Anne Osbourn (2025)},
  year={2025},
  note={arXiv:2507.22514}
}

• arXiv: 2507.22514