Vga Gui Comprehension Eval

vga-gui-comprehension-eval

VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning — Meng et al. (2024) (arXiv:2406.14056, 2024)

What this evaluates

Evaluates a vision-language model's ability to understand graphical user interfaces (GUIs) and answer user questions based on visual content. It specifically probes the model's capacity to avoid hallucinations by grounding responses in actual GUI elements rather than relying solely on textual priors.

Datasets

• GUI Comprehension Bench — total 44; splits: test (44)

Metrics

• GPT evaluation score (primary) — range: percent
  • Automated scoring using ChatGPT to evaluate model responses against expected answers for 44 GUI comprehension questions. Inspired by LLaVA-bench (in-the-wild), responses are scored numerically and aggregated into a final percentage representing overall performance.

Input / output format

Input: A GUI screenshot image (sampled from the Rico dataset) paired with a user question requiring GUI understanding.

Output: Textual response generated by the VLM, which is subsequently passed to ChatGPT for automated scoring.

Scoring recipe

def compute_gpt_evaluation_score(predictions, gold_answers):
    scores = []
    for pred, gold in zip(predictions, gold_answers):
        # ChatGPT evaluates pred against gold based on GUI understanding
        score = call_chatgpt_evaluator(pred, gold)
        scores.append(score)
    return sum(scores) / len(scores) * 100

Common pitfalls

• The evaluation relies entirely on ChatGPT for scoring, which may introduce prompt sensitivity or inconsistency compared to human evaluation.
• The dataset is very small (44 questions from 22 images), limiting statistical significance and generalizability across diverse GUI types.
• Low-resolution evaluation (336x336) shows performance drops but lacks a standardized metric for resolution scaling or token efficiency.

Evidence (verbatim from paper)

Due to the lack of GUI comprehension bench, we sample 22 images from the Rico dataset (excluding training data). And based on these images, we collect 44 user questions which require truly understanding of GUI to response correctly. Inspired by the evaluation method of LLaVA-bench (in-the-wild) Liu et al. ([2024b]), we use ChatGPT to evaluate our model. ... As shown in Table[4], VGA-7b-v1 has shown promising results, achieving the best performance across three separate GPT evaluations. Our model attains a score of 90.83

Citation

@misc{meng2024vga,
  title={VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning},
  author={Meng et al. (2024)},
  year={2024},
  note={arXiv:2406.14056}
}

• arXiv: 2406.14056