Gemini Robotics
Keep Gemini on the perception and planning side of an actuator boundary. The
model may choose a capability; deterministic robot software validates and
executes it.
Choose the endpoint from the interaction
- For a new robot assistant or first simulation demo, read
architect before building a model/robot integration.
It discovers the saved apps checkout and checks compatible examples. If a
baseline was already selected, continue here; existing integration fixes
and API explanations do not need onboarding.
- Use
gemini-robotics-er-2-streaming-preview for a stateful Live API session
that receives text, JPEG frames, or audio and orchestrates robot tools with
low latency.
- Use
gemini-robotics-er-2-preview for discrete embodied-reasoning requests
such as spatial analysis or offline video work. The standard endpoint does
not provide the Live API.
- Treat both model IDs and feature support as preview surfaces. Re-check the
current model overview
before changing dependencies or deployment assumptions.
Make the stream an observe-act-observe loop
- Keep one
client.aio.live.connect session open for the task and run a receive
loop that handles both model content and tool calls.
- Serialize user turns and model-facing heartbeats around unresolved turns and
blocking tools. A text heartbeat is a new reasoning input, not a transport
keepalive, and can interrupt an action as barge-in.
- Declare physical actions with
behavior: BLOCKING. Execute each call through
the robot adapter, then manually return a FunctionResponse with the call ID,
name, and structured result using send_tool_response.
- Stream raw 16-bit, 16 kHz, little-endian PCM for speech input and explicitly
end finite audio with
audio_stream_end=True. Send JPEG camera frames at no
more than the endpoint's current one-frame-per-second limit.
- A camera frame alone updates context but does not trigger reasoning. Pair it
with user audio/text, or use an intentional heartbeat prompt. Heartbeats are
turns and can interrupt generation.
- When a tool exists specifically to observe the world, attach its fresh image
to that call's
FunctionResponse when the SDK supports inline media. This
binds the evidence to the requesting call more deterministically than placing
an unrelated realtime frame immediately before the response.
- The streaming endpoint returns text, not synthesized audio. Route speech
through an independently replaceable TTS adapter or expose speaking as a
bounded tool.
Use Google's current robotics streaming guide
for the volatile SDK syntax. Read FAILURES.md when a session
stalls, ignores images, overlaps actions, or never finishes an audio turn.
Guard the robot outside the model
- Expose semantic capabilities such as named-waypoint navigation, bounded
inspection, or grasping a currently grounded object. Do not expose raw motor
commands, arbitrary poses, or unrestricted coordinates merely because the
function schema can describe them.
- Validate the tool allowlist, exact arguments, ranges, named resources, and
current perception-issued object IDs in ordinary code. A system instruction
and JSON schema improve model behavior but are not the safety boundary.
- Build the advertised tool list from capabilities that passed preflight. Do
not leave a disconnected robot, camera, or accessory visible to the model as
a callable tool.
- Return completion, rejection, and failure states to the model. After motion,
send a fresh observation so the next decision is based on the resulting
scene rather than the pre-action frame.
- Give every long-running action cancellation and a deadline. On session or
tool timeout, invoke the robot's stop/cancel path independently of the model.
- On half-duplex hardware, pause microphone ingestion before speech or another
device action and resume it explicitly afterward. Keep this device handoff
outside the model's control.
- Prove the same semantic contract against a fake adapter, representative
simulation, and finally supervised hardware. Keep simulator- and robot-
specific motion details behind the adapter.
For the evidence behind these choices and their current validation limits, read
SILLY-TURTLEBOT.md for a ROS/Nav2 mobile robot and
STACKCHAN-ER2.md for a USB, audio, camera, and BLE companion.
Use integration for process or transport boundaries, ros2 and navigation
for deterministic mobile-robot execution, and testing for the
fake-to-simulation-to-hardware acceptance ladder.
Done
- A complete user turn can stream input, execute a blocking semantic action,
return its result, and reason from a fresh observation.
- An undeclared or invalid action is rejected before reaching the robot SDK.
- Timeout and cancellation behavior is proven without depending on a model
response.
1---2name: gemini-robotics3description: Integrate and debug Gemini Robotics ER perception, function calls, and guarded execution. For a first natural-language robot assistant demo, start with architect's reference-app selection.4metadata:5 source: https://skillmd.com/skills/robium-ai/gemini-robotics6---78# Gemini Robotics910Keep Gemini on the perception and planning side of an actuator boundary. The11model may choose a capability; deterministic robot software validates and12executes it.1314## Choose the endpoint from the interaction1516- For a new robot assistant or first simulation demo, read17 [architect](../architect/SKILL.md) before building a model/robot integration.18 It discovers the saved apps checkout and checks compatible examples. If a19 baseline was already selected, continue here; existing integration fixes20 and API explanations do not need onboarding.21- Use `gemini-robotics-er-2-streaming-preview` for a stateful Live API session22 that receives text, JPEG frames, or audio and orchestrates robot tools with23 low latency.24- Use `gemini-robotics-er-2-preview` for discrete embodied-reasoning requests25 such as spatial analysis or offline video work. The standard endpoint does26 not provide the Live API.27- Treat both model IDs and feature support as preview surfaces. Re-check the28 current [model overview](https://ai.google.dev/gemini-api/docs/robotics-overview)29 before changing dependencies or deployment assumptions.3031## Make the stream an observe-act-observe loop3233- Keep one `client.aio.live.connect` session open for the task and run a receive34 loop that handles both model content and tool calls.35- Serialize user turns and model-facing heartbeats around unresolved turns and36 blocking tools. A text heartbeat is a new reasoning input, not a transport37 keepalive, and can interrupt an action as barge-in.38- Declare physical actions with `behavior: BLOCKING`. Execute each call through39 the robot adapter, then manually return a `FunctionResponse` with the call ID,40 name, and structured result using `send_tool_response`.41- Stream raw 16-bit, 16 kHz, little-endian PCM for speech input and explicitly42 end finite audio with `audio_stream_end=True`. Send JPEG camera frames at no43 more than the endpoint's current one-frame-per-second limit.44- A camera frame alone updates context but does not trigger reasoning. Pair it45 with user audio/text, or use an intentional heartbeat prompt. Heartbeats are46 turns and can interrupt generation.47- When a tool exists specifically to observe the world, attach its fresh image48 to that call's `FunctionResponse` when the SDK supports inline media. This49 binds the evidence to the requesting call more deterministically than placing50 an unrelated realtime frame immediately before the response.51- The streaming endpoint returns text, not synthesized audio. Route speech52 through an independently replaceable TTS adapter or expose speaking as a53 bounded tool.5455Use Google's current [robotics streaming guide](https://ai.google.dev/gemini-api/docs/robotics-streaming)56for the volatile SDK syntax. Read [FAILURES.md](FAILURES.md) when a session57stalls, ignores images, overlaps actions, or never finishes an audio turn.5859## Guard the robot outside the model6061- Expose semantic capabilities such as named-waypoint navigation, bounded62 inspection, or grasping a currently grounded object. Do not expose raw motor63 commands, arbitrary poses, or unrestricted coordinates merely because the64 function schema can describe them.65- Validate the tool allowlist, exact arguments, ranges, named resources, and66 current perception-issued object IDs in ordinary code. A system instruction67 and JSON schema improve model behavior but are not the safety boundary.68- Build the advertised tool list from capabilities that passed preflight. Do69 not leave a disconnected robot, camera, or accessory visible to the model as70 a callable tool.71- Return completion, rejection, and failure states to the model. After motion,72 send a fresh observation so the next decision is based on the resulting73 scene rather than the pre-action frame.74- Give every long-running action cancellation and a deadline. On session or75 tool timeout, invoke the robot's stop/cancel path independently of the model.76- On half-duplex hardware, pause microphone ingestion before speech or another77 device action and resume it explicitly afterward. Keep this device handoff78 outside the model's control.79- Prove the same semantic contract against a fake adapter, representative80 simulation, and finally supervised hardware. Keep simulator- and robot-81 specific motion details behind the adapter.8283For the evidence behind these choices and their current validation limits, read84[SILLY-TURTLEBOT.md](SILLY-TURTLEBOT.md) for a ROS/Nav2 mobile robot and85[STACKCHAN-ER2.md](STACKCHAN-ER2.md) for a USB, audio, camera, and BLE companion.86Use `integration` for process or transport boundaries, `ros2` and `navigation`87for deterministic mobile-robot execution, and `testing` for the88fake-to-simulation-to-hardware acceptance ladder.8990## Done9192- A complete user turn can stream input, execute a blocking semantic action,93 return its result, and reason from a fresh observation.94- An undeclared or invalid action is rejected before reaching the robot SDK.95- Timeout and cancellation behavior is proven without depending on a model96 response.