Ov Vg Eval

ov-vg-eval

OV-VG: A Benchmark for Open-Vocabulary Visual Grounding — Wang et al. (2023) (arXiv:2310.14374, 2023)

What this evaluates

Evaluates a model's ability to localize objects in images based on natural language descriptions without prior exposure to those specific categories. It probes visual-linguistic alignment, handling of novel vocabulary, and robustness to varying object scales and complex scenes.

Datasets

• OV-VG — total 7272; splits: test (7272); repo https://github.com/cv516Buaa/OV-VG

Metrics

• Acc50 (primary) — range: percent
  • Accuracy at an Intersection over Union (IoU) threshold of 0.5. For each text-image query, a prediction is correct if the IoU between the predicted bounding box and the ground truth bounding box is ≥ 0.5. The final score is the percentage of correctly localized queries across the dataset.

Input / output format

Input: An image and a natural language query describing the target object to be localized.

Output: A single bounding box (typically [x_min, y_min, x_max, y_max] or [x, y, w, h]) representing the predicted location of the target object.

Scoring recipe

correct = 0
for query, pred_box, gt_box in zip(queries, predictions, ground_truths):
    iou = calculate_iou(pred_box, gt_box)
    if iou >= 0.5:
        correct += 1
acc50 = (correct / len(queries)) * 100

Common pitfalls

• Data leakage: Many open-vocabulary baselines are pre-trained on datasets (e.g., COCO, RefCOCO, LVIS) that contain images or categories identical to the test set, artificially inflating performance.
• Small target detection: Objects smaller than 32x32 pixels are frequently missed or mislocalized due to resolution limits and poor visual-linguistic alignment.
• Vocabulary confusion: Complex or ambiguous descriptions lead to predictions of completely unrelated objects.

Evidence (verbatim from paper)

Since the DETR in VLTVG has been pre-trained with BERT, the Acc50 performance revealed in our OV-VG dataset is 2.78%.

Citation

@misc{wang2023ovvg,
  title={OV-VG: A Benchmark for Open-Vocabulary Visual Grounding},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2310.14374}
}

• arXiv: 2310.14374