Matdesign Eval

matdesign-eval

Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents — Kumbhar et al. (2025) (arXiv:2501.13299, 2025)

What this evaluates

Evaluates the ability of LLM agents to generate scientifically grounded, constraint-aligned hypotheses for materials discovery under specific application goals. It probes the model's capacity for iterative refinement, consensus-based validation, and adherence to domain-specific feasibility and novelty criteria.

Datasets

• MatDesign — total ?; splits: test (-1); repo https://github.com/shri071/Hypothesis-Generation-for-Materials-Discovery-and-Design-Using-Goal-Driven-and-Constraint-Guided-LLM

Metrics

• Closeness and Quality (primary) — range: percent
  • A composite evaluation metric assessing hypothesis alignment with the given goal and constraints (Closeness) and six quality dimensions: plausibility, novelty, feasibility, scalability, testability, and impact. Only hypotheses achieving unanimous validation from three independent Critics (CAs) are counted toward the final score.

Input / output format

Input: A goal statement and a set of application-specific constraints for materials discovery.

Output: 20 hypotheses with detailed reasoning per instance.

Scoring recipe

hypotheses = model.generate(goal, constraints, n=20)
validated = []
for h in hypotheses:
    votes = [critic.evaluate(h, goal, constraints) for critic in [CA1, CA2, CA3]]
    if all(votes):  # Unanimous agreement required
        validated.append(h)
# Final metric computed only on unanimously validated hypotheses
score = evaluate_closeness_and_quality(validated)

Common pitfalls

• Hypotheses are only evaluated if they achieve unanimous agreement from all three Critics; partial consensus or majority votes do not count toward the final metric.
• The evaluation is strictly tool-free and relies on LLM-based Critics rather than physical experiments or external databases, which may limit real-world feasibility assessment.
• Iterative refinement is capped at five rounds or stops early upon consensus, which may truncate hypothesis quality if the model requires more iterations to converge.

Evidence (verbatim from paper)

The HGA generates 20 hypotheses with detailed reasoning based on a given goal statement and constraints. These hypotheses are independently reviewed by three CAs, who assess their alignment with the goal and adherence to constraints. Only hypotheses unanimously validated by all three CAs are finalized for evaluation, ensuring reliability through consensus. This configuration introduces an iterative feedback loop involving the HGA, three CAs, and SA to improve the Closeness and Quality of the hypotheses.

Citation

@misc{kumbhar2025hypothesis,
  title={Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents},
  author={Kumbhar et al. (2025)},
  year={2025},
  note={arXiv:2501.13299}
}

• arXiv: 2501.13299