Search3d Lerf Eval

search3d-lerf-eval

OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields — Weijler et al. (2025) (arXiv:2510.21441, 2025)

What this evaluates

Evaluates a model's ability to perform open-vocabulary segmentation and localization in 3D radiance fields, specifically testing its capacity to understand and process hierarchical queries (e.g., object parts relative to whole objects) versus simple object-level queries.

Datasets

• Search3D (adapted) — total ?; splits: test (-1)
• LERF dataset — total ?; splits: test (-1)

Metrics

• mIoU (primary) — range: percent
  • Mean Intersection over Union computed between the predicted segmentation mask and the ground truth annotated masks across all queries.
• accuracy — range: percent
  • Localization accuracy defined as the percentage of queries where the pixel with the highest similarity score falls inside the ground truth bounding box of the target object.

Input / output format

Input: Hierarchical text prompts (e.g., '[OBJ]', '[PART] of a [OBJ]', or compositional phrases like 'leg of the blue chair') provided to a radiance field model for open-vocabulary query.

Output: Predicted segmentation mask and a similarity score map (or bounding box derived from the highest similarity pixel) for each query.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    miou_scores = []
    acc_scores = []
    for query, (pred_mask, sim_map) in predictions.items():
        gt_mask, gt_bbox = ground_truth[query]
        iou = intersection_over_union(pred_mask, gt_mask)
        miou_scores.append(iou)
        top_pixel = argmax(sim_map)
        acc_scores.append(1 if inside_bbox(top_pixel, gt_bbox) else 0)
    return {
        'mIoU': sum(miou_scores) / len(miou_scores) * 100,
        'accuracy': sum(acc_scores) / len(acc_scores) * 100
    }

Common pitfalls

• Models often suffer from a 'bag-of-words' effect, incorrectly segmenting the entire object when queried for a specific part.
• Localization accuracy heavily depends on the similarity score map resolution and the exact definition of the bounding box overlap.
• Hierarchical queries require compositional understanding; simple concatenation of object and part labels often fails without explicit hierarchical modeling.

Evidence (verbatim from paper)

We report results on the task of segmentation using Mean Intersection Over Union (mIoU (Mean Intersection Over Union)) computed between the predicted mask and the annotated masks, as well as performance on the task of localization, quantified as accuracy. For localization, we follow LERF*[[15]]* and consider a true prediction if the pixel with the highest similarity score is inside the bounding box of the object.

Citation

@misc{weijler2025openhype,
  title={OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields},
  author={Weijler et al. (2025)},
  year={2025},
  note={arXiv:2510.21441}
}

• arXiv: 2510.21441