heraldquimpao-creator Gemini Live API development
Use this skill for low-latency, continuous Gemini interactions. Live API code is a sessioned streaming system, not a normal request/response endpoint: connection setup, media pacing, partial responses, interruption, backpressure, reconnection, and cleanup are part of the application design.
Verify the current contract first
Before writing code, use the Gemini Docs MCP to search for the selected Live model, language, transport, media modality, and feature. If the MCP is unavailable, start from:
- Overview:
https://ai.google.dev/gemini-api/docs/live-api - WebSocket quickstart:
https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket - SDK quickstart:
https://ai.google.dev/gemini-api/docs/live-api/get-started-sdk - Capabilities:
https://ai.google.dev/gemini-api/docs/live-api/capabilities - Tool use:
https://ai.google.dev/gemini-api/docs/live-api/tools - Session management:
https://ai.google.dev/gemini-api/docs/live-api/session-management - Ephemeral tokens:
https://ai.google.dev/gemini-api/docs/live-api/ephemeral-tokens - Best practices:
https://ai.google.dev/gemini-api/docs/live-api/best-practices
Treat the Live API as preview or version-sensitive unless current documentation says otherwise. Confirm the model ID, API version, response modalities, supported input formats, session limits, and SDK method names at task time.
Choose the connection approach
Use the Google GenAI SDK when it supports the target language and feature. Use raw WebSockets only when the application needs direct protocol control, a supported SDK does not expose the required behavior, or the task explicitly asks for protocol-level code. Do not mix SDK and raw-WebSocket message formats without checking the current API reference.
Keep API keys off browsers and untrusted clients. For browser or mobile applications, use a backend-issued ephemeral token flow when supported. Use a regular API key only from a trusted server, worker, or local development process that is not distributed to end users.
Establish a session correctly
- Confirm the model and connect to the documented Live API transport.
- Send the required initial setup message before sending media or user input. Include only supported configuration fields.
- Handle the setup acknowledgment and surface setup errors before starting capture.
- Start independent send and receive tasks. The sender reads from user input and applies pacing; the receiver handles every server message and dispatches text, audio, tool calls, turn completion, interruptions, and errors.
- Expose cancellation and close the session in a
finallyblock. Stop microphone, camera, playback, timers, and background tasks on shutdown.
For raw WebSockets, the current protocol uses JSON messages shaped by the BidiGenerateContentClientMessage and BidiGenerateContentServerMessage contracts. The first client message is a setup object. Use the exact message names and nesting from the current API reference rather than copying an old sample.
Stream media deliberately
Document and enforce the media contract before implementation.
| Stream | Design questions to answer from current docs |
|---|---|
| Audio input | PCM or another encoding, sample rate, channel count, bit depth, chunk duration, MIME type, and capture permissions |
| Video input | Supported image MIME type, frame size, frame cadence, downsampling, camera orientation, and bandwidth budget |
| Text input | Whether input is a complete turn or realtime input, turn boundaries, cancellation, and ordering with media |
| Audio output | Output encoding, sample rate, playback buffer, interruption behavior, and conversion required by the device |
Send bounded chunks instead of unbounded buffers. Preserve ordering, measure queue depth, and apply backpressure when the network or playback path falls behind. Avoid blocking the receive loop while decoding, rendering, writing files, or invoking a tool.
Handle turns, VAD, and interruption
Treat partial output as provisional. Render it incrementally, but commit application state only on the documented completion signal. Handle server interruption messages by stopping or fading audio playback, clearing stale queued output, and returning control to the user. When automatic voice activity detection is enabled, test silence, overlapping speech, short utterances, and barge-in; do not assume that local microphone silence maps perfectly to server turn boundaries.
Keep user-visible state explicit: connecting, configuring, listening, thinking, speaking, interrupted, reconnecting, failed, and closed. Do not let a late response from an old session update the UI for a new session.
Session lifecycle and resilience
Plan for session expiration, network loss, mobile backgrounding, device changes, and server errors. Use the current session-management guidance for context transfer, reconnection, compression, or resumption. On reconnect, decide which conversation state can be replayed, which media must be discarded, and how duplicate tool calls are prevented.
Bound memory and latency. Keep audio playback and input queues finite, sample video at a deliberate rate, cancel stale work, and record connection duration, round-trip latency, dropped chunks, reconnect count, turn completion time, and tool duration. Never log raw credentials or unnecessary private audio/video.
Live API tool calls
Treat tool calls received over the session as untrusted requests. Validate arguments, enforce authorization and tenant boundaries in application code, and return compact tool results through the documented Live API message. Add timeouts, cancellation, duplicate-call protection, and confirmation for consequential actions. A tool must not block media reception or audio playback indefinitely.
Testing checklist
Test a text-only session before adding media. Then test one audio input chunk, one video frame, and one complete turn independently. Include malformed setup, unsupported MIME type, permission denial, empty input, slow network, server interruption, reconnect, cancellation during playback, tool timeout, and shutdown while a send or receive task is active.
For deterministic tests, fake the WebSocket and capture devices. For an end-to-end test, use a short bounded session and assert setup success, at least one response event, turn completion or documented equivalent, clean close, and absence of leaked tasks.
Common mistakes
Do not use a normal generateContent loop for a Live API product that needs bidirectional low latency. Do not send media before setup is acknowledged. Do not assume all Live models accept all modalities or output types. Do not concatenate audio chunks without respecting the documented format. Do not run browser API keys in shipped JavaScript. Do not equate a WebSocket connection being open with a healthy session; monitor protocol errors and turn progress.
If the connection fails, classify it as authentication, endpoint or model mismatch, setup schema, media format, rate limit, network, session lifetime, or application backpressure. Search the exact error and current model in the Gemini Docs MCP before changing the protocol.
Minimal architecture
Keep the transport adapter separate from application state:
capture -> bounded input queue -> Live session sender
Live session receiver -> event dispatcher -> UI/playback
\\-> tool executor -> validated result sender
This separation makes it possible to test protocol handling without a microphone or camera and prevents UI work from blocking network reads.