Magicvl 2b Eval

magicvl-2b-eval

MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning — Yi Liu et al. (arXiv:2508.01540, 2025)

What this evaluates

Evaluates a lightweight vision-language model's performance on reasoning, OCR, and real-world understanding benchmarks, alongside deployment efficiency metrics like inference latency and throughput on mobile hardware.

Datasets

• HallusionBench — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• RealworldQA — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• CRPE — total ?; splits: test (-1)
• MME Realworld — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or correctly classified instances across each benchmark.
• ViT latency — range: seconds
  • Wall-clock time in seconds to run the visual encoder inference on the target hardware.
• LLM latency — range: seconds
  • Wall-clock time in seconds to run the language model inference on the target hardware.
• throughput — range: tokens/s
  • Number of generated tokens per second during inference.

Input / output format

Input: Image input paired with a text prompt/question.

Output: Text response generated by the model.

Scoring recipe

# For benchmark accuracy
 correct = 0
 for pred, gold in zip(predictions, gold_labels):
     if pred == gold:
         correct += 1
 accuracy = (correct / len(gold_labels)) * 100

# For deployment metrics
 viT_latency = time_end_vit - time_start_vit
 throughput = total_tokens_generated / total_inference_time

Common pitfalls

• Comparing models across different parameter scales without normalization
• Dynamic resolution changes token count, which may affect latency/throughput comparisons if not accounted for
• On-device latency measurements depend heavily on hardware configuration and background processes

Evidence (verbatim from paper)

As shown in Table[3], MagicVL-2B reduces the total number of tokens by approximately 37.8% (0.52 M vs 0.81 M) during the evaluation, while also achieving improved accuracy (74.5% vs 74.3%).

Citation

@misc{liu2025magicvl2b,
  title={MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning},
  author={Yi Liu et al.},
  year={2025},
  note={arXiv:2508.01540}
}

• arXiv: 2508.01540