Opper Multimodal

Sub-skill of opper — start there for discovery and setup guidance. Source: https://github.com/opper-ai/opper-skills/blob/main/opper-multimodal/SKILL.md

Opper Multimodal & Realtime

Everything beyond text: generate and edit media, run OCR, store and reuse files, send images and PDFs to a model, and run two-way voice over a WebSocket. Same gateway, same Authorization: Bearer $OPPER_API_KEY, same Control Plane governance and tracing as the text endpoints — just different surfaces.

For text generation, chat, structured output, migration from another gateway, and server-side tools (opper:web_search and friends), use the opper-api skill — those ride the compat chat endpoints and stay there.

Concepts: docs.opper.ai/overview/concepts. Multimodal overview: docs.opper.ai/build/multimodal/overview.

The live v3 spec is the source of truth

Default workflow for any question this skill doesn't immediately answer — endpoint, parameter, field, provider knob — grep the spec. Don't guess, don't invent endpoints or model ids. The spec is unauthenticated and definitive:

curl -s https://api.opper.ai/v3/openapi.yaml   # YAML, easier to grep
curl -s https://api.opper.ai/v3/openapi.json   # JSON

Each surface below has its own scoped model-discovery list — call it to learn which models, voices, sizes, durations, and capabilities are live right now. Never hardcode model names.

The surfaces at a glance

One key, one gateway. Pick the endpoint by what you're producing:

How the media endpoints behave

A shared contract across /v3/images, /v3/audio/*, /v3/videos, /v3/ocr:

• model and the prompt/input are owned by Opper. A small set of high-level params is normalized (e.g. size, aspect_ratio, quality, voice, format). Everything you put in parameters is forwarded verbatim to the provider — that's the escape hatch for provider-specific knobs.
• Generated output is stored by default (store: true) to Files and returned with a reusable file_id. Pass store: false to opt out.
• Reuse media without re-uploading by passing a file_id (from a previous generation or a POST /v3/files upload) anywhere a media source is accepted — image / mask / reference_images on images, image / video / reference_images on videos, audio on transcriptions, document on OCR.
• Everything is traced and billed like any other call — visible at platform.opper.ai.

Images — POST /v3/images

# Generate (synchronous)
curl -s -X POST https://api.opper.ai/v3/images \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-image-1", "prompt": "a hot air balloon over green hills", "size": "1024x1024"}'
# → { "data": [{ "url" | "b64_json", "file_id": "file_..." }], "usage": {...} }

Edit / variations: pass a source image (and optional mask or reference_images). Slow models: send "async": true to get a 202 + status_url instead of blocking; poll it like video below.

Audio — speech & transcription

# Text-to-speech
curl -s -X POST https://api.opper.ai/v3/audio/speech \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "openai/tts-1", "input": "Hello from Opper", "voice": "alloy", "format": "mp3"}'
# → { "audio": "<base64>", "file_id": "file_...", "mime_type": "audio/mpeg", "usage": {...} }

# Speech-to-text — audio as file_id, URL, or data-URI
curl -s -X POST https://api.opper.ai/v3/audio/transcriptions \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "openai/whisper-1", "audio": "file_abc123", "language": "en"}'
# → { "text": "...", "language": "en", "duration": 12.3, "segments": [...], "usage": {...} }

GET /v3/audio/models?type=tts lists each speech model's voices, default_voice, and max_length; ?type=stt lists each transcription model's languages, formats, whether it supports diarize (speaker labels — rejected if the model can't do it), and whether it supports stream.

Streaming transcription: send "stream": true to /v3/audio/transcriptions to get incremental transcript.text.delta events over SSE (final transcript.text.done + [DONE]) instead of one blocking response. Only some models support it — check stream in the ?type=stt discovery list; unsupported models return a 400.

Custom voices (cloning): clone a voice from a reference sample, then use its voice_... id anywhere a voice is accepted (e.g. /v3/audio/speech). Voices are scoped to your org/project and provider-agnostic — model picks the backing provider.

# Clone a voice — audio as file_id, URL, or data-URI (reference sample is transient, never stored)
curl -s -X POST https://api.opper.ai/v3/audio/voices \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "mistral/voxtral-mini-2602", "name": "narrator", "audio": "file_abc123"}'
# → { "id": "voice_...", "object": "voice", "provider": "mistral", "expires_at": ..., ... }

# Use the cloned voice
curl -s -X POST https://api.opper.ai/v3/audio/speech \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "mistral/voxtral-mini-2602", "input": "Hello from a cloned voice", "voice": "voice_..."}'

GET /v3/audio/voices lists your voices (with expires_at — some providers expire clones); GET/DELETE /v3/audio/voices/{id} fetch or remove one (delete also removes it at the provider).

Video — POST /v3/videos (asynchronous)

Video is always async: submit, get a 202 with a status_url, then poll until it resolves to a download URL + file_id.

# 1. Submit
curl -s -X POST https://api.opper.ai/v3/videos \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "openai/sora-2", "prompt": "the balloon drifts at dawn"}'
# → 202 { "id": "...", "status_url": "https://api.opper.ai/v3/artifacts/{id}/status" }

# 2. Poll
curl -s -H "Authorization: Bearer $OPPER_API_KEY" \
  "https://api.opper.ai/v3/artifacts/{id}/status"
# → { "status": "completed", "url": "...", "file_id": "file_..." }   (or "processing")

Inspect each model's capabilities via GET /v3/videos/models to tell input modality apart: video_generation is text-to-video, image_to_video needs a source image, video_editing is video-to-video. params.video lists accepted aspect_ratios, resolutions, and max_duration.

OCR — POST /v3/ocr

Turn a PDF or image into structured markdown. Billed per page.

curl -s -X POST https://api.opper.ai/v3/ocr \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "mistral/mistral-ocr-latest",
       "document": {"type": "document_url", "document_url": "https://example.com/report.pdf"}}'
# → { "pages": [{ "index": 0, "markdown": "...", "elements": [...] }], "usage": {"pages_processed": N} }

document accepts a URL, base64 (PDF or image), or a file_id. GET /v3/ocr/models lists OCR models and each one's price_per_page. Guide: docs.opper.ai/build/multimodal/ocr.

Files — /v3/files

General media storage that the generation endpoints write to and read from. Upload once, reference everywhere by file_id — no re-uploading large media across calls.

Per-org byte and file-count quotas apply, enforced on upload and when storing generated outputs. MIME type is sniffed from bytes against a per-purpose allowlist. Guide: docs.opper.ai/build/multimodal/files.

Vision & PDF input (on the chat endpoints)

Sending an image or PDF into a model (rather than generating media) rides the normal compat chat endpoints — it isn't a separate surface. Send image_url / file content parts to a model whose capabilities include vision / pdf:

curl -s "https://api.opper.ai/v3/models?capability=vision"   # which chat models accept images
curl -s "https://api.opper.ai/v3/models?capability=pdf"      # which accept PDFs

Then call POST /v3/compat/chat/completions (or any compat surface) with multimodal content parts — see the opper-api skill for the chat shape. Guide: docs.opper.ai/build/multimodal/vision-pdfs.

Realtime — two-way voice over WebSocket

Low-latency, bidirectional audio (plus text, and image / video frames on some providers) over a WebSocket. Three related endpoints:

Auth — two ways:

• Server-side: connect with a project-scoped runtime Authorization: Bearer $OPPER_API_KEY.
• Browser-side: mint a ticket from POST /v3/realtime-sessions, then connect passing it as ?ticket=<value> or Sec-WebSocket-Protocol: opper-ticket.<value> — never ship the API key to the browser.

# Mint a browser ticket
curl -s -X POST https://api.opper.ai/v3/realtime-sessions \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{"config": {"model": "openai/gpt-realtime-2", "voice": "alloy"}, "locked_fields": []}'
# → { "ticket": "...", "expires_at": "..." }

The first client event is session.start. Turn detection: server_vad (acoustic), semantic_vad (model-based, OpenAI only), or none (client-driven). Providers differ in capability — OpenAI (gpt-realtime-2, vision variant; semantic VAD, image input, tools), Gemini Live (video-frame input, tools), xAI Voice (audio/text, server_vad only), ElevenLabs (conversational voice). Discover live realtime models with GET /v3/models?type=realtime.

Caps & billing: sessions are bounded by per-project concurrency, max duration, and idle timeout; usage flushes every ~30 s with synchronous balance enforcement (a 402 ends the session when the balance runs out). Full event protocol, examples, and billing mechanics: realtime quickstart.

Non-obvious gotchas

• Video is the only always-async media endpoint. POST /v3/videos returns 202 + status_url; poll GET /v3/artifacts/{id}/status. Images, audio, and OCR are synchronous — though /v3/images accepts "async": true for slow models, returning the same poll shape.
• Generated media is stored by default (store: true) and returns a file_id. Pass store: false to skip storage; file_id then won't be reusable.
• Pass file_id, don't re-upload. Any media source field (image, mask, reference_images, audio, video, document) accepts a file_id from a prior generation or a /v3/files upload.
• parameters is a verbatim passthrough. Top-level fields are normalized across providers; anything provider-specific goes in parameters untouched.
• Vision/PDF input is not a media endpoint — it's content parts on a compat chat call to a vision/pdf-capable model. Media generation uses the dedicated endpoints here.
• Realtime browser clients use a ticket, never the API key. Mint it from /v3/realtime-sessions; the key stays server-side.
• Server-side tools (opper:web_search, etc.) live in opper-api, not here. They ride the compat chat endpoints.
• The spec is the most up-to-date reference; this skill follows it. Scoped discovery lists (/v3/images/models, /v3/videos/models, /v3/audio/models, /v3/ocr/models) tell you what's live.

Where to look next