Plantvillagevqa Eval

plantvillagevqa-eval

PlantVillageVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science — Sakib et al. (2025) (arXiv:2508.17117, 2025)

What this evaluates

This benchmark evaluates vision-language models on plant science tasks, ranging from basic species and health identification to detailed symptom verification and higher-order causal or counterfactual reasoning. It probes a model's ability to ground visual attributes, diagnose diseases, and generate descriptive or diagnostic text based on leaf images.

Datasets

• PlantVillageVQA — total 193609; splits: test (-1); HF SyedNazmusSakib/PlantVillageVQA

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard VQA exact-match accuracy: the fraction of questions where the model's predicted answer exactly matches the ground-truth answer (or a predefined set of acceptable answers).

Input / output format

Input: A single RGB image of a plant leaf paired with a natural language question (e.g., binary verification, disease identification, or open-ended diagnostic prompts).

Output: A natural language text response. Answers vary by question type: binary (Yes/No), specific disease names, or open-ended descriptive/diagnostic sentences.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize(pred) == normalize(gold):
            correct += 1
    return correct / len(predictions)

Common pitfalls

• The dataset contains highly imbalanced binary answers (originally 78.6% negative), which can inflate accuracy if models default to 'No' or 'Unhealthy'.
• Open-ended questions require careful normalization or fuzzy matching for scoring, as exact string matches are unlikely without explicit evaluation scripts.
• The dataset lacks predefined train/val/test splits, requiring researchers to define their own evaluation partitions.

Evidence (verbatim from paper)

At this stage, our team of specialists reviewed question variations from all nine categories to ensure scientific accuracy and consistency. We discarded all grammatically incorrect and excessively complex questions. Once validated, we replaced each question template with a randomly selected variation from its paraphrase pool.

Citation

@misc{sakib2025plantvillagevqa,
  title={PlantVillageVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science},
  author={Sakib et al. (2025)},
  year={2025},
  note={arXiv:2508.17117}
}

• arXiv: 2508.17117