Omnilabel Eval

omnilabel-eval

OmniLabel: A Challenging Benchmark for Language-Based Object Detection — Schulter et al. (2023) (arXiv:2304.11463, 2023)

What this evaluates

This benchmark evaluates a model's ability to perform language-based object detection using dynamic, open-vocabulary label spaces. It specifically probes handling of free-form text descriptions, negative examples (descriptions referring to zero objects), and multi-instance references within a single image.

Datasets

• OmniLabel — total 12200; splits: test (-1); repo https://github.com/samschulter/omnilabeltools

Metrics

• harmonic_mean_AP (primary) — range: [0, 1]
  • Compute Average Precision (AP) separately for plain category descriptions and free-form text descriptions. The final score is the harmonic mean of these two AP values. Predictions are matched to ground truth only if bounding boxes overlap sufficiently (IoU) and the prediction index g matches the ground truth description index. One ground truth box can be correctly matched to multiple predictions if they correspond to different descriptions.

Input / output format

Input: An RGB image I_i and a dynamic label space D_i containing a variable number of object descriptions (plain categories and free-form text).

Output: A set of triplets P_i = [(b_i^l, s_i^l, g_i^l)], where b is a 4-coordinate bounding box, s is a confidence score, and g is an integer index linking the prediction to a specific description in D_i.

Scoring recipe

def compute_omnilabel_metric(predictions, ground_truth):
    # Separate descriptions into plain categories and free-form text
    plain_preds, plain_gt = filter_by_type(predictions, ground_truth, 'plain')
    freeform_preds, freeform_gt = filter_by_type(predictions, ground_truth, 'freeform')
    # Compute AP for each group with custom matching (IoU + index g)
    ap_plain = compute_ap_custom(plain_preds, plain_gt, match_fn=match_by_index_and_iou)
    ap_freeform = compute_ap_custom(freeform_preds, freeform_gt, match_fn=match_by_index_and_iou)
    # Harmonic mean of the two APs
    if ap_plain + ap_freeform == 0:
        return 0.0
    return 2 * (ap_plain * ap_freeform) / (ap_plain + ap_freeform)

Common pitfalls

• Standard object detection AP assumes a fixed label space and single-category ground truth; OmniLabel uses a dynamic label space per image and allows one GT box to match multiple predictions via different descriptions.
• Using an arithmetic mean instead of the specified harmonic mean for the final score will incorrectly reward models that excel at one description type while failing at the other.
• Ignoring the prediction index g during matching will break semantic alignment, as the label space is open-vocabulary and descriptions are unique per image.

Evidence (verbatim from paper)

While AP is computed for each category separately (and then averaged) in standard detection, this initial grouping is omitted in OmniLabel. Due to the high specificity of the object descriptions, many of these 'groups' would then consist of only a single object instance in the whole dataset. This can make the metric less robust. However, to ensure that our metric considers the predicted semantic categories, we adjust the matching between prediction and ground truth. While in standard detection the matching is based purely on the bounding boxes via intersection-over-union (since categories are already grouped), we include the index g_i^l that links a prediction with the object descriptions in D_i... We want our metric to give equal importance to both types. Due to the different number of ground truth instances, we first compute AP for both types separately and then take the harmonic mean. Different from the arithmetic mean, the harmonic mean requires good results on both types to achieve a high number on the final metric.

Citation

@misc{schulter2023omnilabel,
  title={OmniLabel: A Challenging Benchmark for Language-Based Object Detection},
  author={Schulter et al. (2023)},
  year={2023},
  note={arXiv:2304.11463}
}

• arXiv: 2304.11463