Symsearch Omnigibson Eval

symsearch-omnigibson-eval

Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search — Mahdi et al. (2026) (arXiv:2603.05642, 2026)

What this evaluates

Evaluates an agent's ability to perform open-vocabulary interactive object search in indoor environments using relational semantic reasoning over 3D scene graphs. It probes exploration efficiency, reasoning accuracy, and computational cost compared to embedding-based and LLM-based planners.

Datasets

• SymSearch — total 200; splits: test (200)
• OmniGibson — total 50; splits: test (50)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Percentage of episodes where the agent successfully locates the target object within the maximum step limit.
• Success weighted by Path Length (SPL) (primary) — range: [0, 1]
  • Standard robotics metric: SR × (optimal_steps / actual_steps). Rewards finding the target efficiently.
• N Steps — range: other
  • Average number of high-level actions taken per episode.
• Inference Time — range: other
  • Average time in seconds spent computing the next action per step.

Input / output format

Input: Textual object query, current 3D scene graph state, and agent observations (RGB-D images or symbolic node features).

Output: Discrete high-level action selection (e.g., navigate to a specific room, frontier, or object node; or interact with a container).

Scoring recipe

def compute_metrics(episodes, optimal_steps):
    sr_sum = 0
    spl_sum = 0
    steps_sum = 0
    inf_time_sum = 0
    for ep in episodes:
        success = 1 if ep.found_target else 0
        sr_sum += success
        spl_sum += success * (optimal_steps / ep.actual_steps) if ep.actual_steps > 0 else 0
        steps_sum += ep.actual_steps
        inf_time_sum += ep.inference_time
    n = len(episodes)
    return {
        'SR': (sr_sum / n) * 100,
        'SPL': spl_sum / n,
        'N_Steps': steps_sum / n,
        'Inference_Time': inf_time_sum / n
    }

Common pitfalls

• Simulation uses oracle 'magic open' actions and ground truth segmentation, which overestimates real-world performance.
• Real-world failures are dominated by perception errors (segmentation/localization) rather than reasoning flaws.
• LLM baselines incur high inference latency and non-determinism, making direct SR comparisons sensitive to sampling variance.
• Inference time is reported separately from total execution time, which includes navigation and manipulation latency.

Evidence (verbatim from paper)

As metrics, we compute the Success Rate (SR) and Success weighted by Path Length (SPL). Additionally, we record the number of high-level steps taken per episode and the average inference time to quantify exploration efficiency.

Citation

@misc{mahdi2026scout,
  title={Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search},
  author={Mahdi et al. (2026)},
  year={2026},
  note={arXiv:2603.05642}
}

• arXiv: 2603.05642