# Ml Committee Uncertainty

> Quantify prediction uncertainty of MACE MLIPs using committee (ensemble) models; flag high-uncertainty structures for DFT verification.

- Skill: `learningmatter-mit/ml-committee-uncertainty` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add learningmatter-mit/ml-committee-uncertainty`
- Raw SKILL.md: https://api.skillmd.com/api/skills/learningmatter-mit/ml-committee-uncertainty/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: learningmatter-mit (https://skillmd.com/u/learningmatter-mit)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/learningmatter-mit/ml-committee-uncertainty

---


# MACE Committee Model Uncertainty Quantification

## Goal
To estimate the epistemic uncertainty of a MACE MLIP by running inference with a committee (ensemble) of independently trained models. Structures where the committee disagrees strongly (high energy or force variance) are flagged as candidates for DFT labelling, supporting active learning workflows and validating MLIP reliability in under-sampled regions of configuration space.

The uncertainty estimate is:
- **Energy uncertainty**: standard deviation of predicted energies across committee members (meV/atom)
- **Force uncertainty**: component-wise force RMSE of the across-committee
  standard deviations (meV/Å): take the sample standard deviation for every
  atom and Cartesian component, then take one root-mean-square over all `3N`
  components. This is the conventional MLIP force-RMSE reduction used by MACE
  reporting.

> [!WARNING]
> **Energy std is only a valid disagreement signal when every committee member shares the same energy reference** — the same training dataset, level of theory, and atomic reference energies (`E0`). If the committee instead pools several *independently pretrained foundation models* (e.g. different MACE-MP / MACE-OMAT / MACE-MATPES releases) rather than same-data/different-seed checkpoints, their absolute energies are not on a common scale: an energy-std ranking will mostly reflect per-model reference-energy offsets, not genuine epistemic disagreement. For this heterogeneous committee flavor, rank structures by **force disagreement** instead — forces are invariant to each model's arbitrary atomic reference energy, so they remain a reliable cross-model signal. See the heterogeneous-committee note under Constraints.

## Instructions

### 1. Obtain Committee Models

A committee requires **N ≥ 3** independently trained MACE checkpoints covering the same chemical system. There are two ways to obtain them:

**Option A — Train with different random seeds (recommended for fine-tuned models)**

Run [ml-mace-finetune](../ml-mace-finetune/SKILL.md) N times, varying only the random seed via the `--seed` flag in `generate_mace_config.py`. Save each checkpoint to a separate directory:

```bash
# Env: mace-agent
# Run for seed=0, seed=1, seed=2 (at minimum)
for SEED in 0 1 2; do
    conda run -n mace-agent python .agents/skills/ml-mace-finetune/scripts/generate_mace_config.py \
        --train-file ./mace_data/train.xyz \
        --valid-file ./mace_data/valid.xyz \
        --model MACE-MH-1 \
        --epochs 200 \
        --lr 1e-4 \
        --batch-size 4 \
        --freeze-backbone \
        --seed ${SEED} \
        --output-dir ./committee_models/seed_${SEED}
    conda run -n mace-agent mace_run_train \
        --config ./committee_models/seed_${SEED}/finetune_config.yaml
done
```

**Option B — Use pre-existing models from the registry**

```bash
mcp_base_search_model_registry(
    chemical_system="Li-Fe-P-O",
    backend="mace",
)
# If multiple models exist for the same system, they can form a committee.
```

### 2. Run Committee Inference

Pass all checkpoint paths to the inference script. It will run each model independently and compute mean ± std across the committee.

```bash
# Env: mace-agent
python .agents/skills/ml-committee-uncertainty/scripts/run_committee_inference.py \
    --structures /path/to/structures_dir_or_file.cif \
    --models ./committee_models/seed_0/mace_finetuned.model \
             ./committee_models/seed_1/mace_finetuned.model \
             ./committee_models/seed_2/mace_finetuned.model \
    --output-dir ./uncertainty_results \
    --energy-threshold 10.0 \
    --force-threshold 200.0
# For MACE-MH foundation models, add:
#   --head omat_pbe
```

**Key parameters:**

| Parameter | Description | Typical value |
|:----------|:------------|:--------------|
| `--structures` | Path to structure file, directory, or `.xyz` trajectory | — |
| `--models` | Space-separated list of MACE checkpoint paths | ≥ 3 models |
| `--energy-threshold` | Flag if energy std > this value (meV/atom) | 5–20 meV/atom |
| `--force-threshold` | Flag if component-wise force RMSE > this value (meV/Å) | 100–300 meV/Å |
| `--output-dir` | Directory for results and plots | — |
| `--device` | `cuda` or `cpu` | `cuda` |
| `--head` | Head name for multi-head models (e.g. `omat_pbe` for MACE-MH-1) | `None` |

### 3. Interpret Results

The script produces:
- `uncertainty_summary.json` — per-structure energy/force mean ± std
- `high_uncertainty_structures/` — `.cif` files of flagged structures requiring DFT
- `uncertainty_distribution.png` — histogram of energy uncertainty across all structures

**Thresholds for DFT flagging:** There is no universal threshold. Empirically:
- Energy std > **10 meV/atom** is a conservative threshold suitable for phonon/stability work
- Energy std > **5 meV/atom** for high-accuracy MD (e.g., melting point, ionic conductivity)
- Force RMSE > **200 meV/Å** generally indicates the configuration is poorly represented in training data

> [!TIP]
> Run this skill on your MD trajectory after a few nanoseconds to check whether the MLIP remains in-distribution throughout the simulation. High uncertainty at late simulation times suggests the trajectory has drifted into unexplored configuration space.

### 4. Send High-Uncertainty Structures to DFT

Pass the flagged structures to the DFT labelling pipeline. See the [mat-sample-pes-by-md](../mat-sample-pes-by-md/SKILL.md) skill for context on when and how to label new structures.

```bash
# High-uncertainty structures needing DFT labels are in:
ls ./uncertainty_results/high_uncertainty_structures/
# → Pass these to atomate2 DFT workflow for labelling
```

### 5. Register Threshold in the Model Registry

After determining an appropriate uncertainty threshold for your system, update the registry so future tasks can apply the same criterion automatically:

```bash
mcp_base_register_model(
    checkpoint_path="./committee_models/seed_0/mace_finetuned.model",
    chemical_system="Li-Fe-P-O",
    backend="mace",
    base_model="MACE-MH-1",
    notes="Committee of 3 models (seed 0/1/2). Use energy_std > 10 meV/atom as DFT flag threshold.",
    tags_json='["battery", "committee"]',
)
```

## Constraints

- **Minimum committee size**: Use at least 3 models. Two models can give misleading std estimates; 5+ models provide more robust uncertainty quantification.
- **Identical architecture**: All committee members must share the same base model architecture and chemical elements (same `--model` flag during fine-tuning). Different architectures cannot be meaningfully ensembled.
- **Same data, different seeds** (fine-tuned committees): When building a committee via Option A above, members should be trained on the same dataset with different random seeds. Mixing training datasets within this committee flavor introduces epistemic uncertainty from data mismatch, not model uncertainty, and confounds the estimate.
- **Heterogeneous / multi-foundation-model committees**: A committee does not have to be same-data/different-seed — a fixed pool of independently pretrained foundation models (e.g. MACE-MP-0, MACE-MP-0b, MACE-MP-0b2, MACE-OMAT-0) is also a valid ensemble for epistemic UQ, and can surface genuine out-of-distribution structures (e.g. an element in an unusual coordination environment) that a same-data seed ensemble would miss. Because these models differ in training data and reference level of theory, however, their **energy** predictions are not on a common scale — always rank this committee flavor by **force disagreement**, never by energy std (see the warning under Goal).
- **Environment**: This script requires the `mace-agent` conda environment.

## References

- Batatia et al., "MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields", *NeurIPS*, 2022.
- Musil et al., "Physics-Inspired Structural Representations for Molecules and Materials", *Chem. Rev.*, 2021. (Committee model UQ)
- Schran et al., "Committee Neural Network Potentials Control Generalization Errors and Enable Active Learning", *J. Chem. Phys.*, 2020.

---

**Author:** Yu Yao
**Contact:** [GitHub @AI4SciDisc](https://github.com/AI4SciDisc)