Visual Prompt Eval

visual-prompt-eval

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want — Lin et al. (2024) (arXiv:2403.20271, 2024)

What this evaluates

Evaluates multimodal large language models' ability to comprehend and reason about visual prompts (points, bounding boxes, free-form shapes) for fine-grained object classification, region captioning, OCR, and complex visual reasoning.

Datasets

• LVIS — total ?; splits: val (-1)
• PACO — total ?; splits: val (-1)
• COCO-Text — total ?; splits: val (-1)
• RefCOCOg — total ?; splits: val (-1)
• MDVP-Bench — total ?; splits: test (-1)
• LLaVA-Bench — total ?; splits: test (-1)
• Ferret-Bench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly classified objects, recognized text, or answered questions in zero-shot settings.
• Semantic Similarity — range: [0, 1]
  • Measures semantic relevance between predicted and ground-truth classifications using word embeddings.
• Semantic IOU — range: [0, 1]
  • Computes the intersection over union of semantic embeddings for predicted and ground-truth regions.
• GPT-4V Score — range: percent
  • Ratio of the model's GPT-4 evaluation score to the GPT-4 baseline score, expressed as a percentage.
• METEOR — range: [0, 1]
  • Metric for evaluating the quality of generated captions against ground truth, considering synonymy and stemming.
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores.

Input / output format

Input: Image paired with a visual prompt (point coordinate, bounding box, or free-form shape) and a text instruction/question targeting the prompted region.

Output: Text response containing the predicted class label, OCR text, region description, or reasoning answer.

Scoring recipe

def evaluate(predictions, gold, metric_name):
    if metric_name == 'Accuracy':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    elif metric_name in ['Semantic Similarity', 'Semantic IOU']:
        return compute_embedding_overlap(predictions, gold)
    elif metric_name == 'GPT-4V Score':
        return (model_gpt4_score / baseline_gpt4_score) * 100
    elif metric_name in ['METEOR', 'CIDEr']:
        return standard_caption_metric(predictions, gold)
    return 0.0

Common pitfalls

• Models are evaluated strictly in zero-shot mode without dataset-specific fine-tuning during testing.
• Visual prompts (boxes/points) are sometimes randomly perturbed or scaled to simulate free-form inputs, which can artificially lower classification metrics if not accounted for.
• GPT-4V scoring for detailed captions uses a ratio to a GPT-4 baseline rather than absolute scores, requiring careful normalization.

Evidence (verbatim from paper)

In all evaluation experiments, we will not continue to fine-tune on a specific dataset but will instead adopt a zero-shot testing approach. Following (Yuan et al., 2024a), we employ two semantic relevance indicators—Semantic Similarity (SS) and Semantic Intersection over Union (S-IOU)—to assess the model's classification performance... GPT-4 is then used to assess the captions generated by the MLLMs, with evaluation scores ranging from 1 to 10 and calculate the ratio of the predicted score to that of GPT-4, expressed as a percentage.

Citation

@misc{lin2024drawandunderstand,
  title={Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2403.20271}
}

• arXiv: 2403.20271