Mobile O Eval

mobile-o-eval

Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device — Shaker et al. (2026) (arXiv:2602.20161, 2026)

What this evaluates

Evaluates a compact on-device unified vision-language-diffusion model's capabilities in multimodal understanding, text-to-image generation, and image editing. It probes the model's ability to align cross-modal representations and generate high-fidelity images while maintaining real-time inference speeds on edge hardware.

Datasets

• GenEval — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)
• SEED — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• ImageEdit — total ?; splits: test (-1)

Metrics

• GenEval overall score (primary) — range: [0, 1]
  • Average accuracy across six categories: Single Object, Two Object, Counting, Colors, Position, and Color Attributes. Scores range from 0 to 1 per category, averaged for the final score.
• Understanding accuracy (%) — range: percent
  • Average accuracy across seven benchmarks (MMMU, MM-Vet, SEED, TextVQA, ChartQA, POPE, GQA). Reported as a percentage.
• ImageEdit score — range: other
  • Composite score measuring edit fidelity and scene preservation on the ImageEdit benchmark.
• Inference latency — range: other
  • Time in milliseconds or seconds to run the vision encoder, compute time-to-first-token (TTFT), and generate an image (20 denoising steps) on specified edge devices.

Input / output format

Input: For understanding: image (resized to 1024×1024) + text prompt/question. For generation: text prompt. For editing: source image + textual editing instruction.

Output: For understanding: text answer. For generation: 512×512 image. For editing: edited 512×512 image.

Scoring recipe

def compute_gen_eval_score(predictions, gold):
    categories = ['Single Obj.', 'Two Obj.', 'Counting', 'Colors', 'Position', 'Color Attri.']
    cat_scores = [sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold) for cat in categories]
    return sum(cat_scores) / len(cat_scores)

def compute_understanding_accuracy(predictions, gold):
    accuracies = []
    for bench_preds, bench_gold in zip(predictions, gold):
        accuracies.append(sum(1 for p, g in zip(bench_preds, bench_gold) if p == g) / len(bench_gold))
    return sum(accuracies) / len(accuracies) * 100

Common pitfalls

• Parameter counts reported in tables include all components (vision encoder, language model, diffusion/UNet), not just the LLM.
• Understanding images are resized to 1024×1024 via bicubic interpolation, while generation operates at 512×512; mixing resolutions will skew results.
• GenEval evaluation strictly uses raw prompts without any post-processing or prompt engineering.

Evidence (verbatim from paper)

Generation quality is measured by GenEval overall score. The proposed post-training stage consistently improves both capabilities.

Citation

@misc{shaker2026mobileo,
  title={Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device},
  author={Shaker et al. (2026)},
  year={2026},
  note={arXiv:2602.20161}
}

• arXiv: 2602.20161