Llava Bench Eval

llava-bench-eval

Visual Instruction Tuning — Haotian Liu et al. (2023) (arXiv:2304.08485, 2023)

What this evaluates

Assesses multimodal chatbot capabilities, including conversation, detailed description, and complex visual reasoning. It measures how well a model follows instructions and understands novel or challenging visual inputs compared to a strong text-only baseline.

Datasets

• LLaVA-Bench — total 150; splits: COCO (90), In-the-Wild (60)

Metrics

• relative_score (primary) — range: percent
  • GPT-4 judges model responses on a 1-10 scale for helpfulness, relevance, accuracy, and detail. Scores are reported as a percentage relative to a text-only GPT-4 baseline that uses ground-truth image captions as visual input.

Input / output format

Input: Image and a textual question (conversation, detailed description, or complex reasoning).

Output: Free-form textual response.

Scoring recipe

baseline_resp = GPT4(question, ground_truth_caption)
model_resp = Model(image, question)
score_1_10 = GPT4_Judge(question, ground_truth_caption, baseline_resp, model_resp)
relative_score = (score_1_10 / 10) * 100

Common pitfalls

• The baseline uses ground-truth captions, giving it an unfair advantage over models that must generate descriptions from scratch.
• LLM-as-a-judge scores can vary with temperature; the paper notes evaluating GPT-4 three times for consistency.
• Benchmarks use out-of-domain or highly challenging images, making direct comparison with standard datasets like COCO difficult.

Evidence (verbatim from paper)

To provide an approximate theoretical upper bound, we create a reference prediction based on the question and the ground-truth textual descriptions, using the text-only GPT-4. ... It evaluates the helpfulness, relevance, accuracy, and level of detail of the responses from the assistants, and gives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. ... We report relative scores w.r.t. the text-only GPT-4 model that uses the textual ground truth description as visual input.

Citation

@misc{liu2023visualinstructiontuning,
  title={Visual Instruction Tuning},
  author={Haotian Liu et al. (2023)},
  year={2023},
  note={arXiv:2304.08485}
}

• arXiv: 2304.08485