graph-robots perceiving-sorting-pairs
A loop-head perception skill for sorting into a visible labelled 2x2 container.
discover_regions runs once: it reads the four printed labels and reconstructs
each compartment floor from calibrated depth. select_pair runs on every loop:
it observes only the described source region, selects one remaining object,
matches its type to one of the perceived labels, segments its graspable part,
and returns its OBB/mask/cloud together with the matching destination OBB. No
object-to-region table or simulator goal state is used.
The identity table comes from the graph
The script knows no object. The graph supplies, as literals on the select
node:
identity_hints-- the visual identities of the labels in play and the TARGET phrase to answer for each (for example "HAMMER is the cyan T-shaped tool and TARGET must be 'blue hammer handle'; ..."). When empty the prompt asks generically for the narrow graspable handle or body.canonical_targets-- a JSON object mapping a label to the exact TARGET phrase; a reply whose TARGET differs for a listed label is rejected. Labels absent from the table accept any TARGET.allowed_labels-- a comma-separated or JSON list restricting the layout to a subset of its labels, for instructions that sort only some categories.
Already-attempted labels are excluded through exclude_label_1..3;
finished is returned when none remain or the model replies DONE.
The reply parser accepts semicolon or newline separators, a missing PIXEL (the box centre stands in), and coordinates given in crop pixels, 0-1000 normalised units, or full-frame pixels -- the first reading that fits the crop wins. Masks under 100 px are rejected as fragments.
The two fixed-camera views are sufficient for layout and object perception. An eye-in-hand camera may improve a downstream grasp skill, but is not required by this skill and is never used to infer the sorting association.