Thyme Multimodal Eval

thyme-multimodal-eval

Thyme: Think Beyond Images — Zhang et al. (2025) (arXiv:2508.11630, 2025)

What this evaluates

Evaluates multimodal large language models on image manipulation, visual perception, mathematical reasoning, and general vision-language tasks. It probes whether autonomous code generation and execution for image processing improves downstream accuracy and reduces hallucination.

Datasets

• MME-RealWorld — total ?; splits: Perception (-1), Reasoning (-1), Overall (-1)
• HR Bench — total ?; splits: FSP (-1), FCP (-1), Overall (-1)
• MathVista — total ?; splits: Mini (-1)
• Hallucination bench — total ?; splits: aAcc (-1), fAcc (-1), qAcc (-1), Overall (-1)
• MMStar — total ?; splits: Overall (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered instances. Computed as (number of correct predictions / total instances) * 100. For multi-part benchmarks, separate accuracy metrics are reported (e.g., FSP/FCP for HR Bench, aAcc/fAcc/qAcc for Hallucination bench, Perception/Reasoning for MME-RealWorld).

Input / output format

Input: Multimodal input consisting of one or more images paired with a text prompt/question requiring visual understanding, code generation, or reasoning.

Output: Textual response, optionally containing executable Python code blocks and simulated sandbox outputs. For benchmarks, the final answer or generated code is extracted for scoring.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
    return (correct / len(golds)) * 100
# Note: For benchmarks like Hallucination, separate metrics (aAcc, fAcc, qAcc) are computed on their respective subsets.
# Code-based tasks may require executing the generated code and comparing sandbox output or final answer.

Common pitfalls

• Failing to report perception and reasoning splits separately for MME-RealWorld and HR Bench.
• Confusing the three Hallucination bench metrics (aAcc, fAcc, qAcc) or averaging them incorrectly.
• Assuming code generation is always required; the model may output direct text, which must be parsed correctly for scoring.
• Overlooking that some benchmarks (e.g., Chart QA, BLINK) have specific validation/test splits that must be matched.

Evidence (verbatim from paper)

We mainly select three categories of benchmarks. The first category focuses on perception tasks because Thyme’s image operations mainly aim to enhance perception ability. These benchmarks include the MME-RealWorld series, HR Bench, V*, RealWorld QA, etc. We report results for different splits of each benchmark. For example, for the MME-RealWorld series, we report perception and reasoning accuracy separately. For HR Bench, we report Fine-grained Single-instance Perception (FSP) and Fine-grained Cross-instance Perception (FCP) separately.

Citation

@misc{zhang2025thyme,
  title={Thyme: Think Beyond Images},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2508.11630}
}

• arXiv: 2508.11630