Drugplayground Eval

drugplayground-eval

DrugPlayGround: Benchmarking Large Language Models and Embeddings for Drug Discovery — Liu et al. (2026) (arXiv:2604.02346, 2026)

What this evaluates

Evaluates LLMs' ability to generate accurate, chemically plausible drug property descriptions and to produce meaningful text embeddings for drug discovery. It probes descriptive accuracy, lexical/structural alignment with ground truth, and embedding similarity for downstream representation tasks.

Datasets

• MolTextNet — total 862; splits: test (862); repo https://github.com/HelloWorldLTY/drugplayground

Metrics

• Normalized Total score (primary) — range: [0, 5]
  • Mean of five individually normalized metrics: BERT, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. Each metric is normalized to a 0–1 scale before averaging, yielding a 0–5 scale.
• Cosine similarity — range: [-1, 1]
  • Average cosine similarity between embeddings of LLM-generated drug descriptions and ground-truth descriptions from MolTextNet. Higher values indicate better semantic alignment.

Input / output format

Input: Drug identifier/name, paired with a prompt template (standard, chain-of-thought, or meta-cognition) and a temperature setting.

Output: Natural language description of the drug's properties, including chemical structure, molecular weight, and pharmacological attributes.

Scoring recipe

def compute_normalized_total(generated_text, reference_text):
    scores = [bert_score(generated, reference),
              rouge1(generated, reference),
              rouge2(generated, reference),
              rougeL(generated, reference),
              bleu(generated, reference)]
    normalized = [normalize_to_0_1(s) for s in scores]
    return sum(normalized) / len(normalized)

Common pitfalls

• Assuming lower temperature always improves generation quality; optimal temperature is highly model-dependent.
• Assuming Chain-of-Thought prompting improves factual accuracy; it often introduces hallucinations and redundant phrasing that lower lexical alignment.
• Equating structured chemical formula outputs with higher factual accuracy; they may still contain incorrect properties.

Evidence (verbatim from paper)

We then compute both schema-driven and structure-driven metrics to quantitatively compare LLM-generated descriptions, produced under different prompt settings, against the ground-truth descriptions. ... compute the average BERT, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU scores, and summarized in Supplementary Figure S1. In addition, we quantify overall performance using a Normalized Total score, defined as the mean of the five individually normalized metrics.

Citation

@misc{liu2026drugplayground,
  title={DrugPlayGround: Benchmarking Large Language Models and Embeddings for Drug Discovery},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.02346}
}

• arXiv: 2604.02346