Tox21 Challenge Eval

tox21-challenge-eval

Measuring AI Progress in Drug Discovery: A Reproducible Leaderboard for the Tox21 Challenge — Ebner et al. (2025) (arXiv:2511.14744, 2025)

What this evaluates

Evaluates molecular toxicity prediction capabilities across diverse AI architectures (descriptor-based models, neural networks, tabular transformers, and zero-shot LLMs) on a standardized chemical safety benchmark.

Datasets

• Tox21 Challenge dataset — total ?; splits: test (-1)

Metrics

• performance (primary) — range: [0, 1]
  • Unspecified in the provided text; typically corresponds to the area under the receiver operating characteristic curve (AUC-ROC) averaged across the 12 Tox21 targets in standard benchmarks.

Input / output format

Input: Molecular structures represented as SMILES strings or tabular molecular descriptors.

Output: Predicted toxicity scores or binary classification labels for each target.

Scoring recipe

def compute_metric(predictions, gold):
    # Metric name unspecified in text; standard Tox21 uses AUC-ROC per target
    scores = []
    for target in targets:
        scores.append(auroc(gold[target], predictions[target]))
    return mean(scores)
# Note: Paper reports median across 5 runs for local evaluation, single run for leaderboard.

Common pitfalls

• Using altered or downstream versions of the dataset (e.g., MoleculeNet, OGB) instead of the original, unaltered Tox21 Challenge dataset.
• Assuming leaderboard scores are deterministic; the paper explicitly notes hardware- and system-level nondeterminism causes discrepancies between local median runs and single-run leaderboard scores.

Evidence (verbatim from paper)

Table[1] summarizes the performance of our baseline toxicity prediction models on the original Tox21-Challenge test set. The results in Table[1] differ slightly from those on the Hugging Face leaderboard. Table[1] reports the median performance across five independent runs conducted on our local infrastructure, whereas the leaderboard shows the score from a single run of the median-performing model, evaluated via the standardized Hugging Face pipeline.

Citation

@misc{ebner2025tox21leaderboard,
  title={Measuring AI Progress in Drug Discovery: A Reproducible Leaderboard for the Tox21 Challenge},
  author={Ebner et al. (2025)},
  year={2025},
  note={arXiv:2511.14744}
}

• arXiv: 2511.14744