Perceiving Object Parts

perceiving-object-parts

Two-step zoom-in perception. The full image gives a small subpart (e.g. a frypan handle is ~3% of pixels) bad signal-to-noise for SAM3 text segmentation; cropping to the parent first brings the subpart up to ~30% of pixels in the cropped image — within SAM3's reliable range.

About parent_prompt and subpart_prompt: they are literal Python strings, NOT subgraph inputs. They are author-time constants per subgraph instance. DO NOT declare them in the subgraph's top-level inputs block, and DO NOT write Ref("in.parent_prompt") or any other Ref(...) for them. Write the strings directly on the inner script node, e.g. "parent_prompt": "frying pan", "subpart_prompt": "long horizontal handle of the frying pan". Only cameras is a flowed subgraph input (wired from the workflow's observation source, identical to perceiving-objects's cameras input).

When to use

• The grasp/place affordance is a part of a larger object (pan handle, drawer pull, moka-pot grip, mug rim, stove burner).
• Plain perceiving-objects with object_name="handle" fails because there are multiple handles in the scene (drawer pull, microwave door, cabinet, ...) and DINO can't disambiguate.

When NOT to use

• The whole object IS the target (perceiving-objects is faster and produces a cleaner OBB).
• The subpart spans the majority of the image already (skip the crop).

Pipeline

observation                       # rgb + depth + intrinsics + camera pose
   │
   ▼ grounding-dino.detect(rgb, parent_prompt)
parent_box (BoundingBox2D)         # broadest of the boxes, or VLM-picked
   │
   ▼ crop_rgb_to_box(parent_box, padding=30)
cropped_image                      # H_new × W_new × 3 uint8
   │
   ▼ grounding-dino.detect(crop, subpart_prompt) → sam3.segment_text
cropped_mask                       # subpart mask in crop coordinates
   │
   ▼ uncrop(cropped_mask → original H × W)
full_mask                          # H × W uint8, zeros outside crop
   │
   ▼ geometry.mask_to_world_points(full_mask, depth, K, T_cam)
world_cloud (PointCloud)
   │
   ▼ geometry.filter_noise → geometry.compute_obb
subpart_obb            # the split calls keep the unfiltered-cloud
                       # fallback when DBSCAN strips too many points

Canonical subgraph layout (mirror perceiving-objects)

{
  "skill": "perceiving-object-parts",
  "inputs": {},
  "nodes": {
    "observe": {
      "type": "tool",
      "tool": "robot.get_observation"
    },
    "perceive_handle": {
      "type": "script",
      "script": "scripts/perceive_subpart.py",
      "inputs": {
        "cameras":        {"$ref": "observe.cameras"},
        "parent_prompt":  "frying pan",
        "subpart_prompt": "long horizontal handle of the frying pan",
        "padding_px": 30
      }
    },
    "found": {"type": "noop"}
  },
  "edges": [
    ["START", "observe"],
    ["observe", "perceive_handle"],
    ["perceive_handle", "found"],
    ["found", "END"]
  ],
  "outputs": {
    "target_obb":   {"$ref": "perceive_handle.obb"},
    "target_mask":  {"$ref": "perceive_handle.mask"},
    "target_cloud": {"$ref": "perceive_handle.cloud"},
    "target_parent_obb":   {"$ref": "perceive_handle.parent_obb"},
    "target_parent_cloud": {"$ref": "perceive_handle.parent_cloud"}
  },
  "exit": {"router_field": null, "success_values": ["found"]},
  "on_error": "not_found"
}

HARD RULE — do NOT add a geometry.filter_and_compute_obb node and bind target_obb to it. Unlike perceiving-objects, this skill's script already returns a clean, noise-filtered OBB in its obb output (computed via geometry.filter_noise + geometry.compute_obb, with a fallback to the unfiltered cloud when DBSCAN strips a thin part below min_points). You MUST bind target_obb directly to {"$ref": "perceive_handle.obb"}. A redundant filter_and_compute_obb node re-filters an already-tiny subpart cloud — DBSCAN on a thin handle shell routinely classifies most of it as noise, collapsing the OBB — and loses the script's unfiltered-cloud fallback.

Key points:

• "inputs": {} — no subgraph-level inputs.
• cameras is produced inside the subgraph by robot.get_observation, identical to perceiving-objects.
• parent_prompt and subpart_prompt are literal strings on the perceive_handle node, NOT subgraph inputs.
• Note the output mask is the PARENT object's mask (used for collision isolation downstream); the subpart's own mask is the subpart_mask output. The obb/cloud outputs ARE the subpart's.