Bee 8b Mllm Eval

bee-8b-mllm-eval

Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs — Yi Zhang et al. (arXiv:2510.13795, 2025)

What this evaluates

Evaluates the visual reasoning, factual accuracy, OCR, chart understanding, and mathematical capabilities of fully open multimodal large language models (MLLMs) against a comprehensive suite of established benchmarks. The protocol tests the model's ability to process images and text prompts, generate responses in a thinking mode, and achieve high scores across general VQA, document/chart analysis, and complex math/reasoning tasks.

Datasets

• Bee-8B Evaluation Benchmarks — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or normalized score based on benchmark-specific rubrics. Some benchmarks (e.g., DocVQA) use LLM-based judging with Qwen3-32B instead of exact string matching.

Input / output format

Input: Multimodal input consisting of an image and a text prompt/question.

Output: Text response (answer or step-by-step reasoning trace).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        # Use exact match or LLM-judge depending on benchmark
        if is_correct(pred, gold_ans):
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Some benchmarks use LLM-as-a-judge (e.g., DocVQA) instead of exact string matching.
• Evaluation is run in 'thinking mode' with a 16,384 token output limit.
• Scores are reported as percentages; some benchmarks have different normalization schemes.

Evidence (verbatim from paper)

Its most significant advantages are observed in factual accuracy and complex multi-step reasoning, directly reflecting the strengths of our Honey-Data-15M dataset. We evaluated our model using a customized VLMEvalKit*(Duan et al., [2024])*... Our model was evaluated with the thinking mode and a maximum response length of 16,384 tokens.

Citation

@misc{zhang2025bee,
  title={Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs},
  author={Yi Zhang et al.},
  year={2025},
  note={arXiv:2510.13795}
}

• arXiv: 2510.13795