Md Evalbench Eval

md-evalbench-eval

MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamics — Zhuofan Shi et al. (2026) (arXiv:2601.02075, 2026)

What this evaluates

Evaluates large language models on molecular dynamics domain knowledge, LAMMPS scripting syntax comprehension, and automatic generation of executable LAMMPS simulation scripts from natural language instructions.

Datasets

• MD-EvalBench — total ?; splits: test (-1); repo https://github.com/FredericVAN/PKU_MDAgent2

Metrics

• Exec-Success@$k$ (primary) — range: percent
  • Proportion of tasks for which at least one of the k generated candidates can be successfully executed in LAMMPS. Calculated as the number of tasks with at least one successful execution divided by the total number of tasks.
• Code Human Score — range: [0, 10]
  • Average subjective rating on a scale of [0, 10] assigned by domain experts based on readability, robustness, and physical correctness of the generated scripts.
• QA Total Score — range: percent
  • Aggregated performance score across single-choice, multiple-choice, fill-in-the-blank, and short-answer questions, reported as a percentage or point score.

Input / output format

Input: Natural language questions for QA tasks; natural language task descriptions for code generation tasks.

Output: Text answers for QA tasks; executable LAMMPS script code for code generation tasks.

Scoring recipe

def score_qa(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def score_code_exec(predictions, k=3):
    success_count = 0
    for task_preds in predictions:
        top_k = task_preds[:k]
        if any(run_lammps(candidate) for candidate in top_k):
            success_count += 1
    return success_count / len(predictions) * 100

def score_code_human(predictions):
    ratings = [expert_rating(p) for p in predictions]
    return sum(ratings) / len(ratings)

Common pitfalls

• Assuming code generation success without actual LAMMPS execution validation
• Confusing the three sub-datasets (MD-KnowledgeEval, LAMMPS-SyntaxEval, LAMMPS-CodeGenEval) under the unified MD-EvalBench suite
• Treating the subjective Code Human Score as an objective metric without noting the [0,10] expert rating scale

Evidence (verbatim from paper)

We adopt two primary metrics. Exec-Success@$k$ measures the proportion of tasks for which at least one of the k generated candidates can be successfully executed in LAMMPS. Code Human Score is a subjective rating in the range [0,10] assigned by domain experts based on readability, robustness, and physical correctness.

Citation

@misc{shi2026mdagent2,
  title={MDAgent2: Large Language Model for Code Generation and Knowledge Q&A in Molecular Dynamics},
  author={Zhuofan Shi et al. (2026)},
  year={2026},
  note={arXiv:2601.02075}
}

• arXiv: 2601.02075