Made Eval

made-eval

MADE: Benchmark Environments for Closed-Loop Materials Discovery — Malik et al. (2026) (arXiv:2601.20996, 2026)

What this evaluates

Evaluates end-to-end autonomous materials discovery pipelines by measuring how effectively different policies (planners, generators, selectors, and agentic orchestrators) can find thermodynamically stable compounds under constrained oracle query budgets. It probes the trade-offs between discovery efficiency, structural diversity, and adaptivity as chemical complexity and stability thresholds increase.

Datasets

• MADE Benchmark Environments — total 30; splits: evaluation (30); repo https://github.com/diffractivelabs/MADE

Metrics

• AF (Acceleration Factor) (primary) — range: other
  • Not explicitly defined in the text; reported as an averaged metric over episodes and system sizes to quantify discovery speed relative to baselines.
• EF (Enhancement Factor) — range: other
  • Not explicitly defined in the text; reported alongside AF to measure efficiency gains of adaptive strategies over static baselines.
• AUDC (Area Under Discovery Curve) — range: other
  • Not explicitly defined in the text; measures cumulative discovery performance over the query budget.
• mSUN (mean Stable Unique Number) — range: other
  • Not explicitly defined in the text; counts the mean number of unique stable structures discovered per episode.
• Mean pairwise L1 distance — range: other
  • Composition-level diversity metric measuring the average L1 distance between discovered compositions.
• Number of unique space-groups — range: other
  • Structure-level diversity metric counting distinct crystallographic space groups among discovered stable structures.

Input / output format

Input: Initial set of structures H0 from Materials Project (MP-20), oracle query budget (50), stability threshold (default 0.1 eV/atom), and iterative feedback of previously explored compositions and formation energies.

Output: Sequence of proposed crystal structures/compositions until the oracle budget is exhausted, culminating in a set of discovered stable structures.

Scoring recipe

def score_episode(policy, initial_structures, budget=50, threshold=0.1):
    discovered = []
    history = []
    for step in range(budget):
        proposal = policy.select_next(history)
        energy = oracle.evaluate(proposal)
        history.append((proposal, energy))
        if energy <= threshold:
            discovered.append(proposal)
    return discovered, history

af, ef, audc, msun = [], [], [], []
for system in systems:
    for episode in range(5):
        disc, hist = score_episode(policy, system.H0)
        af.append(compute_af(disc, baseline_disc))
        ef.append(compute_ef(disc, baseline_disc))
        audc.append(compute_audc(hist))
        msun.append(len(set(disc)))
return mean(af), mean(ef), mean(audc), mean(msun)

Common pitfalls

• Surrogate model (MLIP) ranking performance degrades significantly at tighter stability thresholds (e.g., 0.01 eV/atom) due to errors near the convex hull.
• Results are averaged over only 10 randomly sampled chemical systems per space, which may not fully represent the combinatorial complexity of high-entropy alloys.
• Fixed oracle budget of 50 queries per episode limits evaluation of long-horizon discovery strategies beyond this horizon.

Evidence (verbatim from paper)

We report averaged metrics over all episodes and system sizes for each discovery policy using a random generator as a baseline in Figure [3] and Table [1]. The Chemeleon + MLIP pipeline achieves the highest AF among non-agentic methods (AF = 6.4) and the largest AUDC, consistent with prior work demonstrating the effectiveness of surrogate screening in materials discovery. The fully agentic LLM orchestrator achieves discovery efficiency comparable to the strongest modular pipelines, with significantly improved enhancement factor (EF = 6.0) and competitive AUDC and mSUN (Table[1]).

Citation

@misc{malik2026made,
  title={MADE: Benchmark Environments for Closed-Loop Materials Discovery},
  author={Malik et al. (2026)},
  year={2026},
  note={arXiv:2601.20996}
}

• arXiv: 2601.20996