Tracking Objects

tracking-objects

Long-running tracker skill. Init on first frame, update per tick, close on exit. Used as a parallel sibling to other long-running skills (e.g. a policy) when continuous state estimation is needed.

The skill is class-based and stateful: the tracker session id and the last good mask/box live on the skill instance, so repeated visits to the same state within one workflow execution can resume the session instead of re-seeding it (pass close_on_exit: false to keep the session open across visits; the final visit — or the instance teardown — closes it).

It is also a streaming skill (gap.streaming: true): each update tick publishes a tracker snapshot {mask, box, confidence, object_present, n_updates} via ctx.publish, so downstream {"$ref": "<node>"} consumers see the latest tracked state while the loop is still running.

Install

Depends on the sam3 tool bundle:

uv sync --extra sam3   # (pip: pip install -e "open-robot-skills[sam3]")

When to use

• A workflow that needs the live mask + box of an object across many frames (e.g., a supervisor that monitors a target's location while a policy manipulates it).
• Wrapped under a parallel state with a join_policy so the tracker is cooperatively cancelled when the sibling branch finishes (the loop checks ctx.cancel_token every tick).

Output

Returns the final mask, box, confidence, and a flag indicating whether the object was visibly present at exit. Intermediate updates are published as streaming snapshots; the return value exposes only the final state.

Tool form

The bundle also exposes the loop as a flat tool — tracking-objects.track — for callers that want to invoke it as a single unit (one fresh tracker session per call) rather than as a workflow state.