LLM Downscaling Eval

llm-downscaling-eval

Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models — Mark Endo, Serena Yeung-Levy (2025) (arXiv:2511.17487, 2025)

What this evaluates

Evaluates how reducing the size of the language model component in multimodal models impacts task performance, specifically isolating and measuring the bottlenecks in visual perception versus logical reasoning across multiple benchmarks.

Datasets

• Grounding — total ?; splits: test (-1)
• NIGHTS — total ?; splits: test (-1)
• PieAPP — total ?; splits: test (-1)
• OCR-VQA — total ?; splits: test (-1)
• Fine-grained Perception — total ?; splits: test (-1)
• Logical Reasoning — total ?; splits: test (-1)
• Math — total ?; splits: test (-1)
• Science & Technology — total ?; splits: test (-1)

Metrics

• performance (primary) — range: [0, 1]
  • Task performance measured as standard accuracy or benchmark score; reported as relative performance dropoff when downscaling the LLM or perception module.

Input / output format

Input: Image(s) and text prompt/question. For the decoupled analysis, the perception module first extracts question-specific visual details offline, which are then inserted as text instructions into the reasoning module prompt.

Output: Textual response answering the question or completing the task.

Scoring recipe

def calculate_performance(predictions, gold):
    # Task-specific scoring (e.g., exact match, F1, or regression score)
    score = compute_task_metric(predictions, gold)
    return score

# Report relative performance dropoff:
# dropoff = (baseline_score - downscaled_score) / baseline_score

Common pitfalls

• Assuming reasoning capacity is the primary bottleneck; the evaluation shows perception downscaling causes larger performance drops across most tasks.
• Ignoring the impact of captioning training on perception modules (e.g., LLaVA-OneVision), which can artificially mitigate perceived bottlenecks on in-domain data.
• Failing to decouple perception and reasoning, which masks the distinct failure modes of each module when scaled down.

Evidence (verbatim from paper)

We present plots showing the performance dropoff from LLM downscaling across all evaluated tasks in Figure[A1]. As described in the main text, most tasks exhibit minimal performance decline when downscaling the LLM, except for a handful of vision-centric tasks that exhibit substantially larger drops (e.g., Grounding, NIGHTS, PieAPP).

Citation

@misc{endo2025downscaling,
  title={Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models},
  author={Mark Endo, Serena Yeung-Levy (2025)},
  year={2025},
  note={arXiv:2511.17487}
}

• arXiv: 2511.17487