Chemo Eval

chemo-eval

ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025 — Xu et al. (2025) (arXiv:2511.16205, 2025)

What this evaluates

Evaluates multimodal large language models' ability to solve Olympiad-level theoretical chemistry problems requiring visual perception, chemical reasoning, and structured problem-solving. It specifically probes the visual perception bottleneck in chemistry tasks and tests the effectiveness of multi-agent orchestration and structured visual enhancement.

Datasets

• ChemO — total 9; splits: test (9)

Metrics

• normalized rubric-based score (primary) — range: [0, 100]
  • Raw points from a rubric-based grader (built from official IChO rubrics) are mapped to a global 100-point scale using original problem weights. The total score is the sum of weighted contributions across all 9 problems.
• LLM-as-a-Judge similarity — range: [0, 1]
  • An external LLM judge evaluates the semantic alignment between the model response and the reference solution, outputting a score in [0,1]. Higher values indicate stronger agreement in content and reasoning.

Input / output format

Input: AER reformulated problem P_AER containing text and images, optionally augmented with structured visual guidance G (e.g., SMILES extracted by OCSR).

Output: Answers for all sub-questions, typically in symbolic or textual formats (e.g., SMILES) that can be automatically graded or compared against reference solutions.

Scoring recipe

def compute_metrics(predictions, problems, rubrics, reference_solutions):
    total_norm_score = 0.0
    for pred, prob, rubric, ref in zip(predictions, problems, rubrics, reference_solutions):
        raw = rubric_grader(pred, rubric)
        total_norm_score += (raw / prob.original_points) * prob.normalized_max
    similarity = external_llm_judge(predictions, reference_solutions)
    return total_norm_score, similarity

Common pitfalls

• The benchmark uses Assessment-Equivalent Reformulation (AER) to convert visual outputs to machine-readable formats; failing to apply AER makes automatic grading impossible.
• The normalized score is a weighted sum based on original IChO point allocations mapped to a 100-point scale, not a simple average across problems.
• LLM-as-a-Judge similarity is a supplementary metric for semantic alignment and should not be used as the primary performance indicator.

Evidence (verbatim from paper)

Evaluation Metrics. Tab. 2 reports two metrics for each problem in the format normalized rubric-based score and LLM-as-a-Judge similarity: Normalized rubric-based score. For each problem, the rubric-based grader returns the total points obtained under the unified deductive framework built from the official IChO rubrics. These raw points are mapped to a global 100-point scale using the original weights. The row Original Points lists the raw allocations for P1-P9, which sum to 385. The row Norm. shows the maximum normalized contribution of each problem on this scale. LLM-as-a-Judge similarity. In parallel, we compute an LLM-as-a-Judge similarity score in [0,1] using an external LLM judge that evaluates the semantic alignment between the model response and the reference solution.

Citation

@misc{xu2025chemlabs,
  title={ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2511.16205}
}

• arXiv: 2511.16205