Diningbench Eval

diningbench-eval

DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain — Jin et al. (2026) (arXiv:2604.10425, 2026)

What this evaluates

This benchmark evaluates Vision-Language Models on fine-grained visual discrimination, nutritional quantification from images, and complex food-related visual question answering. It probes the models' ability to fuse multi-view imagery, perform volumetric reasoning, and avoid parametric knowledge biases when identifying dishes and estimating macronutrients.

Datasets

• DiningBench — total ?; splits: test (-1); repo https://github.com/meituan/DiningBench

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of total instances. Calculated as the number of exact matches between model output and ground truth divided by the total number of samples.
• MAPE (primary) — range: percent
  • Mean Absolute Percentage Error. Computed as the average of |(actual - predicted) / actual| across all samples, typically expressed as a percentage to quantify regression error in nutrition estimation.

Input / output format

Input: One to four images of a dish (multi-view) paired with a text prompt. Prompts vary by task: classification options, nutrition estimation queries, or open-ended VQA questions.

Output: Text response containing either a selected dish name, numerical nutritional values, or a natural language answer. Models may optionally generate chain-of-thought reasoning before the final prediction.

Scoring recipe

def score(predictions, golds, task_type):
    if task_type in ['classification', 'vqa']:
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
        return correct / len(golds)
    elif task_type == 'nutrition':
        mape_vals = []
        for p, g in zip(predictions, golds):
            pred = extract_numeric(p)
            gold = extract_numeric(g)
            if gold != 0:
                mape_vals.append(abs(pred - gold) / gold)
        return (sum(mape_vals) / len(mape_vals)) * 100

Common pitfalls

• Chain-of-Thought prompting often degrades performance on fine-grained classification and nutrition regression tasks by introducing noise or hallucination instead of refining predictions.
• Smaller models suffer from information overload when provided with 3+ multi-view images, causing performance to plateau or drop rather than improve.
• Models frequently default to parametric knowledge priors (e.g., guessing common dishes) rather than relying on visual evidence, especially for long-tail or visually similar dishes.

Evidence (verbatim from paper)

Quantifying nutritional content from visual cues proves to be the most demanding task. Even the state-of-the-art Gemini-3-Pro-Preview yields an Average MAPE of 24.45%, reflecting a non-negligible margin of error. The challenge is more pronounced for GPT-4o, which suffers from a high error rate of 42.43%.

Citation

@misc{jin2026diningbench,
  title={DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain},
  author={Jin et al. (2026)},
  year={2026},
  note={arXiv:2604.10425}
}

• arXiv: 2604.10425