qhjqhj00 spectrumbench-eval
SpectrumWorld: Artificial Intelligence Foundation for Spectroscopy — Zhuo Yang et al. (2025) (arXiv:2508.01188, 2025)
What this evaluates
Evaluates multimodal large language models on spectroscopy tasks spanning signal processing, perception, semantic understanding, and molecular generation. It probes the models' ability to align cross-modal data, reason over spectral patterns, and generate accurate chemical structures or spectra.
Datasets
- SpectrumBench — total ?; splits: test (-1); repo https://github.com/little1d/SpectrumLab
Metrics
accuracy (%)(primary) — range: percent- For multiple-choice tasks, correct answers score 1 and incorrect answers score 0. For generation tasks, a dedicated scoring model (GPT-4o) evaluates the output against the ground truth, normalizing the score between 0 and 1. Non-conforming predictions receive a score of 0. The final metric is the average accuracy across all tasks.
Input / output format
Input: Multiple-choice questions with four options for signal/perception/semantic tasks; molecular structures/SMILES for Molecule-to-Spectrum generation; multiple spectral images for Spectrum-to-Molecule generation.
Output: Selected option for multiple-choice tasks; generated spectrum or molecule structure for generation tasks. Predictions must strictly conform to the specified output format, otherwise they are scored 0.
Scoring recipe
def evaluate(predictions, golds, task_types):
scores = []
for pred, gold, task in zip(predictions, golds, task_types):
if task == "multiple_choice":
scores.append(1.0 if pred == gold else 0.0)
elif task == "generation":
if not matches_format(pred):
scores.append(0.0)
else:
scores.append(gpt4o_score(pred, gold))
return sum(scores) / len(scores)
Common pitfalls
- Generation task predictions that do not strictly follow the required output format are automatically assigned a score of 0.
- Evaluation of generation tasks relies on an external LLM (GPT-4o) for scoring, which may introduce non-determinism or bias compared to exact-match metrics.
- The benchmark explicitly requires models to generate reasoning trajectories to prevent data leakage, making format compliance and reasoning quality critical for scoring.
Evidence (verbatim from paper)
For signal-, perception-, and semantic-level tasks, SpectrumBench standardizes them into a multiple-choice question format, with each question having four options. A correct answer is scored as 1, and an incorrect answer is scored as 0. Generation-level tasks usually do not have fixed-form answers. For Molecule-to-Spectrum tasks, the input is a molecule, and the output is a spectrum. For Spectrum-to-Molecule tasks, the input consists of multiple spectral images, and the output is a molecule. We aim to encourage models to generate meaningful reasoning trajectories rather than simply providing a final answer. This approach can help circumvent the issue of data leakage. Therefore, we use an additional MLLM to score the responses following these steps: (1) Model predictions that do not conform to the specified output format for a given question are assigned a score of zero. (2) For predictions meeting the required format, a dedicated scoring model evaluates the model’s output against the answer, assigning a score normalized between 0 and 1. GPT-4o is employed as the scoring model in our experiment. This design standardizes the primary evaluation metric across all tasks in SpectrumBench
Citation
@misc{yang2025spectrumworld,
title={SpectrumWorld: Artificial Intelligence Foundation for Spectroscopy},
author={Zhuo Yang et al. (2025)},
year={2025},
note={arXiv:2508.01188}
}
- arXiv: 2508.01188