Multimodal Multiplication Eval

multimodal-multiplication-eval

Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs — Balter et al. (2026) (ACL Findings (2026), 2026)

What this evaluates

This benchmark evaluates the arithmetic computation capabilities of multimodal LLMs by testing their ability to multiply numbers presented across different input modalities (text, images, audio) and representations (numerical vs. alphabetic). It isolates computational difficulty from perceptual factors by systematically varying digit length and sparsity.

Datasets

• HDS Benchmark — total 144; splits: test (144)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Binary exact-match accuracy of the model's generated answer against the ground truth product. Computed as the fraction of correctly solved problems across the test split.
• 50% C threshold — range: other
  • The arithmetic load value C (total digits × non-zero digits) at which the fitted logistic regression curve predicts 50% Accuracy. Higher values indicate greater robustness to difficulty.

Input / output format

Input: Multiplication problems formatted as '{num1} × {num2} = ?' rendered as plain text, rendered images, or audio recordings, using either numerical digits or alphabetic words.

Output: The model must generate the exact numerical product (forced completion or standard generation).

Scoring recipe

def evaluate(predictions, golds, loads):
    correct = [1 if p == g else 0 for p, g in zip(predictions, golds)]
    acc = sum(correct) / len(correct)
    # Fit logistic regression: logit(p) = beta0 + beta1 * load
    # 50% C threshold = -beta0 / beta1
    # R^2 = fraction of outcome variability explained by the fit
    return {'accuracy': acc, '50pct_C': -beta0/beta1, 'R2': r2}

Common pitfalls

• Over-interpreting the degradation slope (beta1) without accounting for the baseline intercept (beta0); models with steep slopes can still outperform flat-slope models at moderate loads due to higher starting accuracy.
• Assuming heuristic-specific LoRA adapters improve arithmetic performance; the evaluation shows they successfully steer reasoning traces but generally degrade correctness compared to the base model.
• Treating modality effects as uniform penalties; the protocol shows modality differences primarily manifest as intercept shifts rather than consistently steeper degradation curves.

Evidence (verbatim from paper)

Figure 2 shows logistic regression curves for the probability of a correct answer versus arithmetic load across modalities. Accuracy declines monotonically with load across modalities, but the ordering depends on representation and model. The 50% load threshold is especially useful for comparing models because it combines the intercept and slope into a single difficulty scale.

Citation

@misc{balter2026multiplication,
  title={Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs},
  author={Balter et al. (2026)},
  year={2026},
  note={ACL Findings (2026)}
}

• arXiv: 2604.18203