V3det Eval

v3det-eval

V3Det: Vast Vocabulary Visual Detection Dataset — Jiaqi Wang et al. (2023) (arXiv:2304.03752, 2023)

What this evaluates

Probes an object detector's ability to localize and classify instances across a vast, hierarchical vocabulary of over 13,000 categories. It evaluates both closed-set detection performance and open-vocabulary generalization to novel, unseen categories.

Datasets

• V3Det — total 243038; splits: train (183354), val (29821), test (29863)

Metrics

• AP (primary) — range: [0, 1]
  • Mean Average Precision computed across IoU thresholds from 0.5 to 0.95, following the COCO evaluation protocol. It averages the AP at each IoU threshold.

Input / output format

Input: RGB image with associated ground-truth bounding boxes and category labels. For open-vocabulary evaluation, text descriptions of base and novel categories are also provided.

Output: List of predicted bounding boxes, each containing a class label, confidence score, and bounding box coordinates (x, y, width, height).

Scoring recipe

def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    predictions = sorted(predictions, key=lambda x: x['score'], reverse=True)
    tp, fp = [], []
    for pred in predictions:
        gt_match = find_best_match(pred, ground_truth, iou_thresh)
        if gt_match and not gt_match['used']:
            tp.append(1); gt_match['used'] = True
        else:
            fp.append(1)
    tp_cum = np.cumsum(tp)
    fp_cum = np.cumsum(fp)
    precision = tp_cum / (tp_cum + fp_cum + 1e-8)
    recall = tp_cum / (len(ground_truth) + 1e-8)
    return compute_ap_from_curve(precision, recall)
# Final AP is mean of AP across IoU thresholds 0.50:0.05:0.95

Common pitfalls

• The test split images are not publicly released; researchers must upload predictions to a designated public test server for evaluation.
• Open-vocabulary benchmarks require strict separation of base ($C_{base}$) and novel ($C_{novel}$) classes, reporting $bAP$ and $nAP$ separately rather than a single aggregate score.
• The dataset contains 13,204 categories with hierarchical inclusion relationships, so standard flat-category detectors may struggle without specific adaptations like the Norm Linear Layer or repeat factor sampler.

Evidence (verbatim from paper)

We follow the evaluation metrics of the COCO dataset to report the mean average precise (AP) on different IoU thresholds (i.e., 0.5 ~ 0.95).

Citation

@misc{wang2023v3det,
  title={V3Det: Vast Vocabulary Visual Detection Dataset},
  author={Jiaqi Wang et al. (2023)},
  year={2023},
  note={arXiv:2304.03752}
}

• arXiv: 2304.03752