Fgbench Eval

fgbench-eval

MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery — Kuznetsov et al. (2026) (arXiv:2603.03517, 2026)

What this evaluates

Tests functional-group reasoning capability by asking models to predict how molecular properties change when specific functional groups are added, removed, or modified at given positions.

Datasets

• FGBench — total ?; splits: test (-1)

Metrics

• Accuracy (Acc) (primary) — range: [0, 1]
  • Fraction of correctly predicted boolean (True/False) answers for functional group modification questions.
• RMSE — range: other
  • Root Mean Squared Error between predicted and actual numeric property values.
• Validity (Val) — range: [0, 1]
  • Fraction of predictions that are chemically valid and properly formatted.

Input / output format

Input: Reference molecule and a textual description of a functional group modification (e.g., adding/removing groups at specified positions).

Output: Chain-of-thought reasoning in tags, followed by an block containing a boolean label or numeric value.

Scoring recipe

preds = []
for rep in range(num_reps):
    out = model.generate(sample_prompt(), mol, aug_mol)
    preds.append(parse_answer(out))
final = mode(preds) if boolean else median(preds)
acc = sum(p == gold for p in final) / len(final)
rmse = sqrt(mean((p - g)**2 for p, g in zip(final, gold)))
val = sum(is_valid(p) for p in final) / len(final)

Common pitfalls

• Numeric predictions require careful parsing of the block; models often output units or text alongside numbers.
• Boolean tasks are evaluated via majority vote across stochastic runs, not single greedy decoding.
• Functional group positions must be correctly mapped to the molecular graph before evaluation.

Evidence (verbatim from paper)

We calculate accuracy (Acc) for the binary tasks and RMSE for regression. We report accuracy Acc for questions, RMSE for regression, and fraction of valid predictions (Val).

Citation

@misc{kuznetsov2026mmai,
  title={MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery},
  author={Kuznetsov et al. (2026)},
  year={2026},
  note={arXiv:2603.03517}
}

• arXiv: 2603.03517