Openfungraph Eval

openfungraph-eval

Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces — Zhang et al. (2025) (arXiv:2503.19199, 2025)

What this evaluates

Evaluates a model's ability to predict open-vocabulary functional 3D scene graphs from posed RGB-D images. It specifically probes the detection of objects and interactive elements, as well as the inference of their functional relationships (e.g., switch controls light) in real-world indoor spaces.

Datasets

• SceneFun3D — total ?; splits: val (8), test (12)
• FunGraph3D — total ?; splits: test (-1)

Metrics

• Recall@K (primary) — range: [0, 1]
  • Node recall is the fraction of ground-truth nodes successfully retrieved within top-K, where success requires non-zero 3D IoU and CLIP embedding cosine similarity ranking. Triplet recall requires all three components (object, element, relationship) to be retrieved in top-K. Node association and edge prediction are derived from triplet recall counts.

Input / output format

Input: Posed RGB-D images of indoor environments.

Output: Functional 3D scene graph containing nodes (objects and interactive elements) with 3D coordinates and open-vocabulary labels, and edges representing functional relationships between them.

Scoring recipe

def compute_recall(pred_items, gt_items, K, sim_fn):
    retrieved = 0
    for gt in gt_items:
        if any(sim_fn(pred, gt) > 0 and rank(pred, gt) <= K for pred in pred_items):
            retrieved += 1
    return retrieved / len(gt_items)

# Node recall uses 3D IoU > 0 and CLIP cosine similarity
node_recall = compute_recall(pred_nodes, gt_nodes, K=3 or 10, sim_fn=clip_cosine)

# Triplet recall requires all 3 components retrieved in top-K
triplet_recall = compute_recall(pred_triplets, gt_triplets, K=5 or 10, sim_fn=bert_cosine)
node_assoc = triplet_recall / len(gt_triplets)
edge_pred = triplet_recall / node_assoc

Common pitfalls

• Baselines using ground-truth instance segmentation for graph inference are unfair; must use predicted nodes for fair comparison.
• Triplet evaluation is strict: all three components (object, interactive element, relationship) must be individually retrieved within top-K.
• Relationship retrieval uses BERT embeddings for open-vocabulary predicates, not CLIP.

Evidence (verbatim from paper)

To evaluate open-vocabulary functional 3D scene graphs effectively, a new quantitative metric is essential. Existing approaches, such as ConceptGraph*[[27]], rely on subjective human assessments, while Open3DSG*[[41]] approaches evaluation as a label retrieval task, assuming all ground-truth nodes are known, an assumption that diverges from our real-world setting. To address this, we extend the Open3DSG Recall@K metric*[[41]] with a node detection component, using spatial overlap between predicted and ground-truth nodes, inspired by evaluation techniques on 2D scene graph generation*[[106], [89], [50], [87], [88]].

Citation

@misc{zhang2025openfungraph,
  title={Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.19199}
}

• arXiv: 2503.19199