Cc Cliff Eval

cc-cliff-eval

Less can be more for predicting properties with large language models — Alampara et al. (2024) (arXiv:2406.17295, 2024)

What this evaluates

Evaluates whether large language models can effectively learn and utilize spatial coordinate information versus categorical/compositional data for property prediction. It quantifies the systematic performance degradation (the 'Coordinate-Category Cliff') when tasks require geometric reasoning rather than simple type matching, and tests whether scaling model size or dataset volume mitigates this deficit.

Datasets

• Synthetic coordinate-category datasets — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/lamalab-org/MatText
• Materials property datasets (shear modulus, bulk modulus, perovskite formation energy) — total ?; splits: 5-fold CV (-1); repo https://github.com/lamalab-org/MatText

Metrics

• CC-Cliff (primary) — range: other
  • Computed as L_coord,α - L_cat,α. A value of zero indicates equal performance on coordinate vs category tasks; a positive value indicates systematic underperformance on coordinate-dependent tasks.
• RMSE — range: other
  • Root Mean Square Error between predicted and ground-truth scalar property values. Lower values indicate better predictive performance.

Input / output format

Input: Text representations of material structures (e.g., SLICES, Cif P1, Local-Env) containing atom types, 3D coordinates, bonds, and symmetry information, tokenized for LLM input.

Output: Predicted scalar property value (e.g., energy, modulus) or categorical label.

Scoring recipe

def compute_cc_cliff(losses_at_alpha):
    alpha_g = [0.0, 0.2, 0.4]
    alpha_c = [0.6, 0.8, 1.0]
    L_coord = sum(losses_at_alpha[a] for a in alpha_g) - 3 * math.log(0.5)
    L_cat = sum(losses_at_alpha[a] for a in alpha_c) - 3 * math.log(0.5)
    return L_coord - L_cat

def compute_rmse(predictions, targets):
    return math.sqrt(mean((p - t)**2 for p, t in zip(predictions, targets)))

Common pitfalls

• Assuming that scaling model parameters or dataset size will resolve the coordinate learning deficit; the paper explicitly shows scaling has negligible effect on RMSE.
• Interpreting the CC-Cliff aggregation formula literally as log(α) without recognizing it represents aggregated log-loss values at specific α settings.
• Expecting that adding explicit geometric information to text representations will improve performance; the evaluation shows geometry-aware representations often perform worse or equal to composition-only ones due to LLM architectural biases.

Evidence (verbatim from paper)

To quantify how well the models learn from each type of information, we compute two aggregate error metrics: the coordinate contribution (L_coord,α) and the category contribution (L_cat,α). ... If L_coord,α > L_cat,α, we call this phenomenon the Coordinate-Category Cliff (CC-Cliff)—the systematic underperformance on tasks requiring coordinate information. The coordinate contribution and category contribution are defined as: L_coord,α = ∑{α∈α_g}log(α) - 3×log(0.5) and L_cat,α = ∑{α∈α_c}log(α) - 3×log(0.5), where α_g = {0,0.2,0.4} and α_c = {0.6,0.8,1.0}.

Citation

@misc{alampara2024lesscanbemore,
  title={Less can be more for predicting properties with large language models},
  author={Alampara et al. (2024)},
  year={2024},
  note={arXiv:2406.17295}
}

• arXiv: 2406.17295