Icrl Molecular Eval

icrl-molecular-eval

Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning — Zhang et al. (2025) (arXiv:2509.17552, 2025)

What this evaluates

Evaluates whether text-based LLMs can effectively leverage high-dimensional non-text modality representations (e.g., molecular embeddings from foundation models) via training-free in-context learning, comparing various representation injection and projection strategies.

Datasets

• ESOL — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
• Caco_wang — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
• AqSolDB — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
• LD50_Zhu — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL
• AstraZeneca — total ?; splits: test (1000); repo https://github.com/ztlmememe/LLMxFM_ICRL

Metrics

• RMSE (primary) — range: other
  • Root Mean Square Error, calculated as the square root of the mean squared difference between predicted and true values. Lower values indicate better performance.
• Pearson r — range: [0, 1]
  • Pearson’s correlation coefficient measuring the linear correlation between predicted and true values. Higher values indicate better performance.

Input / output format

Input: Textual SMILES sequences and/or high-dimensional molecular representations formatted as in-context examples (r_i, y_i) or (x_i, r_i, y_i) fed to Llama-3.1-70B-Instruct.

Output: Predicted continuous molecular property value (regression target).

Scoring recipe

def compute_metrics(preds, targets):
    rmse = np.sqrt(np.mean((preds - targets) ** 2))
    pearson_r = np.corrcoef(preds, targets)[0, 1]
    return {'RMSE': rmse, 'Pearson_r': pearson_r}

Common pitfalls

• Text-level PCA injection relies on large context windows and degrades when combined with raw text features.
• Zero-padding and random projection fail to generate suitable representations, often performing worse than random noise baselines.
• Projector initialization and activation functions have minimal impact on performance, contrary to assumptions about complex projection layers.

Evidence (verbatim from paper)

This subsection evaluates various ICRL methods across different scenarios, analyzing their strengths and limitations. The conclusions we proposed are consistent across different metrics, i.e., Pearson’s correlation coefficient (Pearson r) and Root Mean Square Error (RMSE).

Citation

@misc{zhang2025icrl,
  title={Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.17552}
}

• arXiv: 2509.17552