Perceiving Objects Oneshot

perceiving-objects-oneshot

Single VLM call over a set-of-marks overlay. Pipeline:

observe → perceive → filter_obb

perceive runs:

1. grounding-dino.detect with a broad object. text prompt
2. One vlm.query showing the image with letter-labeled boxes: "Which letter is the ? Reply with one letter or 'none'."
3. On none: emit found: False so the subgraph exits not_found. On a letter: sam3.segment_box on the chosen box, geometry.mask_to_world_points for the cloud.

When to use

• Clean-all-items / multi-item loops where the cycle needs a clean "no match" signal to terminate via target.not_found → done.
• Tasks where the target description is generic ("any item on the floor", "the next remaining grocery item") rather than a specific scene-spec id.
• Uncluttered scenes with distinct, reasonably sized targets where the set-of-marks letter pick is reliable.

When NOT to use

• Small / cluttered targets (< 40 px wide) — prefer perceiving-objects whose pairwise crop tournament is far more reliable in that regime.

Recommended subgraph state flow

3 states: observe → perceive → filter_obb (mirrors perceiving-objects).

State details:

About object_name below: it is a literal Python string — the natural noun phrase for the object you are perceiving, drawn from this subgraph's description (e.g. "alphabet soup", "basket", "any grocery item on the floor"). It is a constant per subgraph instance, NOT a binding. DO NOT write Ref("in.object_name") or any other Ref(...); the coordinator does not declare object_name as a subgraph input. Write the string directly, e.g. "object_name": "any grocery item on the floor". The same rule applies to object_description if you set it.

1. observe — type: tool, tool: "robot.get_observation", inputs: {}. Connector tool; flat name only.
2. perceive — type: script, file scripts/<sg>/perceive_simple.py from this bundle. Inputs: cameras=Ref("observe.cameras"), object_name="<noun phrase from the subgraph description>", plus any optional literals (object_description, dino_prompt). Returns {found, cloud, mask, score}. When the VLM picks "none" or DINO emits no detections, found is False and the downstream filter_obb step then raises (empty cloud) — caught by the subgraph's on_error: "not_found" exit.
3. filter_obb — type: tool, tool: "geometry.filter_and_compute_obb", inputs={"points": Ref("perceive.cloud")}. Returns {"obb": <OrientedBoundingBox>}.

Wiring the exit (HARD)

Linear perceive → filter_obb → found → END. The filter_obb tool raises on empty clouds (the not-found path), and the subgraph's on_error: "not_found" catches that. Do NOT add conditional edges on perceive — the linear path plus set_on_error is sufficient.

sg.add_node("filter_obb", type="tool",
            tool="geometry.filter_and_compute_obb",
            inputs={"points": Ref("perceive.cloud")})
sg.add_exit("found")
sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)
sg.set_on_error("not_found")

Bind the subgraph outputs (ALL THREE — required, no exceptions):

sg.set_outputs(
    target_obb=Ref("filter_obb.obb"),
    target_mask=Ref("perceive.mask"),
    target_cloud=Ref("perceive.cloud"),
)

Note that geometry.filter_and_compute_obb returns {"obb": ...}, so the OBB binding walks into the obb field (Ref("filter_obb.obb"), NOT a bare Ref("filter_obb")). See references/geometry_calling_conventions.md.