AI Core/media Generation

Media Generation

Dependency note: This skill builds on ai-core. Read it first for critical rules.

All media activities (image, speech, transcription, video) follow the same server/client architecture: a generate*() function on the server, an SSE transport via toServerSentEventsResponse(), and a framework hook on the client.

Setup -- Image Generation End-to-End

Server (API route or TanStack Start server function)

// routes/api/generate/image.ts
import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

export async function POST(req: Request) {
  const { prompt, size, numberOfImages } = await req.json()

  const stream = generateImage({
    adapter: openaiImage('gpt-image-1'),
    prompt,
    size,
    numberOfImages,
    stream: true,
  })

  return toServerSentEventsResponse(stream)
}

Client (React)

import { useGenerateImage, fetchServerSentEvents } from '@tanstack/ai-react'
import { useState } from 'react'

function ImageGenerator() {
  const [prompt, setPrompt] = useState('')
  const { generate, result, isLoading, error, reset } = useGenerateImage({
    connection: fetchServerSentEvents('/api/generate/image'),
  })

  return (
    <div>
      <input
        value={prompt}
        => setPrompt(e.target.value)}
        placeholder="Describe an image..."
      />
      <button
        => generate({ prompt })}
        disabled={isLoading || !prompt.trim()}
      >
        {isLoading ? 'Generating...' : 'Generate'}
      </button>

      {error && <p>Error: {error.message}</p>}

      {result?.images.map((img, i) => (
        <img
          key={i}
          src={img.url || `data:image/png;base64,${img.b64Json}`}
          alt={img.revisedPrompt || 'Generated image'}
        />
      ))}

      {result && <button
    </div>
  )
}

TanStack Start: Server Function Streaming (recommended)

When using TanStack Start, return toServerSentEventsResponse() from a server function. The client fetcher receives a Response and the hook parses it as SSE automatically:

// lib/server-functions.ts
import { createServerFn } from '@tanstack/react-start'
import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'
import type { OpenAIImageModel } from '@tanstack/ai-openai'

export const generateImageStreamFn = createServerFn({ method: 'POST' })
  .inputValidator((data: { prompt: string; model?: OpenAIImageModel }) => data)
  .handler(({ data }) => {
    return toServerSentEventsResponse(
      generateImage({
        adapter: openaiImage(data.model ?? 'gpt-image-1'),
        prompt: data.prompt,
        stream: true,
      }),
    )
  })

import { useGenerateImage } from '@tanstack/ai-react'
import { generateImageStreamFn } from '../lib/server-functions'

function ImageGenerator() {
  const { generate, result, isLoading } = useGenerateImage({
    fetcher: (input) => generateImageStreamFn({ data: input }),
  })

  return (
    <button
      => generate({ prompt: 'A sunset over mountains' })}
      disabled={isLoading}
    >
      {isLoading ? 'Generating...' : 'Generate'}
    </button>
  )
}

Core Patterns

1. Image Generation

Supported adapters: openaiImage (dall-e-2, dall-e-3, gpt-image-1, gpt-image-1-mini, gpt-image-2), geminiImage (gemini-3.1-flash-image, gemini-3.1-flash-lite-image, gemini-3-pro-image, imagen-4.0-generate-001, etc.) and byteplusImage (Seedream — seedream-4-0-250828, seedream-4-5-251128, the 5.0 family).

Use the GA Gemini image ids. gemini-3.1-flash-image-preview and gemini-3-pro-image-preview were shut down on 2026-06-25 and now 404. They survive in the type union only as deprecated aliases so existing code keeps compiling — a call to them typechecks and then fails at runtime. Use gemini-3.1-flash-image / gemini-3-pro-image instead.

Seedream quirks: watermark defaults to true (pass modelOptions: { watermark: false } for a clean image), size is a token ('1K' | '2K' | '4K') or explicit '2048x2048' pixels but never a mix, and numberOfImages is an upper bound — Seedream has no n, so it maps onto group-image mode and the model may return fewer. Reads ARK_API_KEY.

import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'
import { geminiImage } from '@tanstack/ai-gemini'

// OpenAI with quality/background options
const openaiResult = await generateImage({
  adapter: openaiImage('gpt-image-1'),
  prompt: 'A cat wearing a hat',
  size: '1024x1024',
  numberOfImages: 2,
  modelOptions: {
    quality: 'high',
    background: 'transparent',
    output_format: 'png',
  },
})

// Gemini native model with aspect-ratio sizes
const geminiResult = await generateImage({
  adapter: geminiImage('gemini-3.1-flash-image'),
  prompt: 'A futuristic cityscape at night',
  size: '16:9_4K',
})

// Gemini Imagen model
const imagenResult = await generateImage({
  adapter: geminiImage('imagen-4.0-generate-001'),
  prompt: 'A landscape photo',
  modelOptions: { aspectRatio: '16:9' },
})

Result shape: ImageGenerationResult with images array where each entry has b64Json?, url?, and revisedPrompt?. OpenAI image URLs expire after 1 hour -- download or display immediately.

Image-conditioned generation: multimodal prompt parts

Both generateImage() and generateVideo() accept the prompt either as a plain string or as an ordered array of content parts (TextPart / ImagePart / VideoPart / AudioPart — the same shapes used elsewhere in TanStack AI). Part order is meaningful: natively multimodal providers (Gemini, OpenRouter) receive parts in order; named-field providers (OpenAI, fal, xAI) extract media parts and flatten the text. Prompt text is always sent verbatim — to reference inputs from the prompt, write the provider's own syntax (fal @Image1, OpenAI "image 1" prose); the SDK never injects or rewrites markers. Each media part may carry an optional metadata.role hint that adapters use to route the part to the provider-specific field. The accepted part types are narrowed per model at compile time via the adapter's input-modality map.

import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

// Image-to-image (OpenAI gpt-image-2 / gpt-image-1, dall-e-2)
await generateImage({
  adapter: openaiImage('gpt-image-2'),
  prompt: [
    { type: 'text', content: 'Turn this into a cinematic product photo' },
    { type: 'image', source: { type: 'url', value: 'https://…/product.png' } },
  ],
})

// Multi-reference (up to 16 for gpt-image models; up to ~14 for Gemini native
// — a provider limit, not enforced by the SDK)
await generateImage({
  adapter: openaiImage('gpt-image-2'),
  prompt: [
    { type: 'text', content: 'Apply the second image as style to the first' },
    { type: 'image', source: { type: 'url', value: 'https://…/product.png' } },
    { type: 'image', source: { type: 'url', value: 'https://…/style.png' } },
  ],
})

// Inpaint via metadata.role === 'mask' (OpenAI gpt-image models, dall-e-2; fal mask_url)
await generateImage({
  adapter: openaiImage('gpt-image-2'),
  prompt: [
    { type: 'text', content: 'Replace the masked region with a tree' },
    { type: 'image', source: { type: 'url', value: 'https://…/photo.png' } },
    {
      type: 'image',
      source: { type: 'url', value: 'https://…/mask.png' },
      metadata: { role: 'mask' },
    },
  ],
})

// Image-to-video (OpenAI Sora: single input_reference; fal: image_url + optional
// end_image_url; OpenRouter: frame_images + input_references)
import { generateVideo } from '@tanstack/ai'
import { falVideo } from '@tanstack/ai-fal'

await generateVideo({
  adapter: falVideo('fal-ai/kling-video/v3/pro/image-to-video'),
  prompt: [
    { type: 'image', source: { type: 'url', value: 'https://…/first.png' } },
    { type: 'text', content: 'Slow cinematic push-in' },
    {
      type: 'image',
      source: { type: 'url', value: 'https://…/last.png' },
      metadata: { role: 'end_frame' },
    },
  ],
})

URL inputs that require an upload throw by default. Most adapters pass a type: 'url' source straight through to the provider. Three paths can't — OpenAI images.edit(), OpenAI Sora input_reference, and Gemini Veo — because the provider only accepts uploaded bytes (Veo also takes a gs:// reference). For those, an HTTP(S) URL would have to be downloaded and buffered in memory, which can OOM constrained runtimes, so they throw on an HTTP(S) URL image input by default. Pass a data: URI (or gs:// for Veo), or opt in with allowUrlFetch: true on the adapter config (createOpenaiImage(model, apiKey, { allowUrlFetch: true }), and likewise on createOpenaiVideo / createGeminiVideo). data: URIs never need the flag.

Role hints (metadata.role):

Provider support matrix:

Video and audio prompt parts follow the same metadata.role convention for video-to-video and lipsync flows on fal. Grok accepts one source video part on grok-imagine-video with modelOptions.mode: 'edit' | 'extend' and rejects audio parts. Other providers throw when those parts are passed.

2. Audio Generation (Music, Sound Effects)

Distinct from TTS — generateAudio() produces non-speech audio content. Supported adapters: geminiAudio (Lyria 3 Pro / Lyria 3 Clip) and falAudio (MiniMax Music, DiffRhythm, Stable Audio, ElevenLabs SFX, etc.).

import { generateAudio } from '@tanstack/ai'
import { falAudio } from '@tanstack/ai-fal'

const result = await generateAudio({
  adapter: falAudio('fal-ai/diffrhythm'),
  prompt: 'An upbeat electronic track with synths',
  duration: 10,
})

// result.audio.url or result.audio.b64Json (provider-dependent)
// result.audio.contentType e.g. "audio/mpeg"

Client hook:

import { useGenerateAudio, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, isLoading } = useGenerateAudio({
  connection: fetchServerSentEvents('/api/generate/audio'),
})

// Trigger: generate({ prompt: 'Upbeat synths', duration: 10 })
// Play:    <audio src={result.audio.url} controls />

3. Text-to-Speech

Adapters: openaiSpeech (tts-1, tts-1-hd, gpt-4o-audio-preview) and byteplusSpeech (seed-audio-1.0).

BytePlus Seed Speech is a separate product from ModelArk — it reads BYTEPLUS_VOICE_API_KEY, not ARK_API_KEY, and an Ark key there fails with 45000010 Invalid X-Api-Key. Output is capped at 120 seconds. There is no top-level speaker field — voice is sent as references: [{ speaker }], and modelOptions.references replaces that array rather than merging, so passing references for voice cloning silently drops voice. Voice ids ending _uranus_bigtts are TTS 2.0, _mars_bigtts / _moon_bigtts are TTS 1.0, and *_emo_v2_* are the 1.0 voices that accept emotion tags. Formats: wav, mp3, pcm, ogg_opus; watermark is also available on modelOptions.

import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

const result = await generateSpeech({
  adapter: openaiSpeech('tts-1-hd'),
  text: 'Hello, welcome to TanStack AI!',
  voice: 'alloy', // alloy | echo | fable | onyx | nova | shimmer | ash | ballad | coral | sage | verse
  format: 'mp3', // mp3 | opus | aac | flac | wav | pcm
  speed: 1.0, // 0.25 to 4.0
})

// result.audio is base64-encoded audio
// result.format is the output format string
// result.contentType is the MIME type (e.g. "audio/mpeg")

Client hook:

import { useGenerateSpeech, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, isLoading } = useGenerateSpeech({
  connection: fetchServerSentEvents('/api/generate/speech'),
})

// Trigger: generate({ text: 'Hello!', voice: 'alloy' })
// Play:   <audio src={`data:audio/${result.format};base64,${result.audio}`} controls />

4. Audio Transcription

Adapters: openaiTranscription (whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize) and byteplusTranscription (seed-asr — synchronous, no polling; audio up to 2 hours / 100 MB; also reads BYTEPLUS_VOICE_API_KEY).

Capturing audio in the browser: Use useAudioRecorder from @tanstack/ai-react to record directly in the browser, then pass the recording as the audio input to generate(), or use recording.part as a prompt part in chat/generation calls. No transcoding or extra dependencies required — the recorder returns the native browser format (audio/webm or audio/mp4). For transcription, wrap it as a data: URL so the provider gets the real content type; passing raw recording.base64 makes the adapter assume audio/mpeg and mislabel the webm/mp4 bytes.

import {
  useAudioRecorder,
  useTranscription,
  fetchServerSentEvents,
} from '@tanstack/ai-react'

function VoiceNote() {
  const { isRecording, start, stop } = useAudioRecorder()
  const { generate } = useTranscription({
    connection: fetchServerSentEvents('/api/transcribe'),
  })

  async function finish() {
    const recording = await stop()
    const mimeType = recording.mimeType.split(';')[0] // strip ;codecs=...
    await generate({ audio: `data:${mimeType};base64,${recording.base64}` })
  }

  return (
    <button ? finish : start}>
      {isRecording ? 'Stop & transcribe' : 'Record'}
    </button>
  )
}

// routes/api/transcribe.ts
import { generateTranscription, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiTranscription } from '@tanstack/ai-openai'

export async function POST(request: Request) {
  // The client hook below posts { data: { audio: dataUrl, language } }
  const { audio, language } = (await request.json()).data

  const stream = generateTranscription({
    adapter: openaiTranscription('whisper-1'),
    audio, // File, Blob, base64 string, or data URL
    language,
    responseFormat: 'verbose_json',
    modelOptions: {
      timestamp_granularities: ['word', 'segment'],
    },
    stream: true,
  })

  // On the client, result.text is the transcript, result.language the
  // detected language, result.duration the seconds, result.segments the
  // timestamped segments (word-level timestamps are in result.words).
  return toServerSentEventsResponse(stream)
}

For speaker diarization, use openaiTranscription('gpt-4o-transcribe-diarize'). When no response format is given it defaults the request to response_format: 'diarized_json' and chunking_strategy: 'auto' (a top-level responseFormat of 'json'/'text' opts out of speaker segments); do not pass prompt, include, or timestamp_granularities with this model.

Client hook:

import { useTranscription, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, isLoading } = useTranscription({
  connection: fetchServerSentEvents('/api/transcribe'),
})

// Trigger: generate({ audio: dataUrl, language: 'en' })

5. Video Generation (Experimental -- async polling)

Video generation uses a jobs/polling architecture. The server creates a job, polls for status, and streams updates to the client. Adapters: openaiVideo (Sora), geminiVideo (Veo / Omni Flash), grokVideo, byteplusVideo (Seedance), falVideo (Kling, MiniMax, Hunyuan, …), and openRouterVideo (OpenRouter's dedicated POST /api/v1/videos gateway — Seedance, Veo, Wan, Kling, Sora 2 Pro and others through one API key; getVideoJobStatus() returns the video as a data: URL since OpenRouter's download URLs require the API key, and surfaces the gateway-reported cost as usage.cost).

import {
  generateVideo,
  getVideoJobStatus,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { openaiVideo } from '@tanstack/ai-openai'

// Non-streaming: manual polling loop
const { jobId } = await generateVideo({
  adapter: openaiVideo('sora-2'),
  prompt: 'A golden retriever playing in sunflowers',
  size: '1280x720',
  duration: 8,
})

let status = await getVideoJobStatus({ adapter: openaiVideo('sora-2'), jobId })
while (status.status !== 'completed' && status.status !== 'failed') {
  await new Promise((r) => setTimeout(r, 5000))
  status = await getVideoJobStatus({ adapter: openaiVideo('sora-2'), jobId })
}

// Streaming: server handles polling, client gets real-time updates
export async function POST(request: Request) {
  const { prompt } = await request.json()
  const stream = generateVideo({
    adapter: openaiVideo('sora-2'),
    prompt,
    stream: true,
    pollingInterval: 3000,
    maxDuration: 600_000,
  })
  return toServerSentEventsResponse(stream)
}

Google Veo (@tanstack/ai-gemini) uses the same jobs/polling flow. Its duration option is typed per model (4 | 6 | 8 for the Veo 3.1 models); use adapter.snapDuration(seconds) to coerce raw seconds and adapter.availableDurations() to enumerate the valid set. Image prompt parts route by metadata.role: first un-roled / 'start_frame' image → input image, 'end_frame' → lastFrame, 'reference' / 'character' → referenceImages:

import { generateVideo } from '@tanstack/ai'
import { geminiVideo } from '@tanstack/ai-gemini'

const adapter = geminiVideo('veo-3.1-generate-preview')
adapter.availableDurations() // { kind: 'discrete', values: [4, 6, 8] }

const { jobId } = await generateVideo({
  adapter,
  prompt: 'A golden retriever playing in sunflowers',
  size: '16:9', // Veo sizes are aspect ratios: '16:9' | '9:16'
  duration: adapter.snapDuration(7), // 6
  modelOptions: { resolution: '1080p', generateAudio: true },
})
// Note: Veo result URLs require the Google API key to download
// (x-goog-api-key header or ?key= query parameter).

Gemini Omni Flash (geminiVideo('gemini-omni-1.1-flash')) is served by the Interactions API instead of Veo's operations flow — same adapter, routed by model. duration is any number of seconds in the 3–10 range (fractional ok, default 10 — availableDurations() reports the range), size is an aspectRatio_resolution template ('16:9' or '16:9_1080p'; suffix '360p' | '720p' | '1080p' | '4k', default 720p), and the finished video arrives inline as a data:video/mp4;base64,… URL (no key needed to use it). Image/video prompt parts are sent as interaction content blocks, grouped as images, then videos, then text (no metadata.role routing); data sources go inline, url sources pass through as-is (never downloaded — use Gemini Files API URIs for remote media). For conversational editing, pass a prior generation's jobId as modelOptions.previous_interaction_id with a prompt describing the change. gemini-omni-flash-preview remains a deprecated alias until it shuts down on 2026-09-30.

import { generateVideo } from '@tanstack/ai'
import { geminiVideo } from '@tanstack/ai-gemini'

const omni = geminiVideo('gemini-omni-1.1-flash')
const first = await generateVideo({
  adapter: omni,
  prompt: 'A violinist outdoors',
})
// …poll first.jobId to completion, then edit it:
const edited = await generateVideo({
  adapter: omni,
  prompt: 'Make the violin invisible',
  modelOptions: { previous_interaction_id: first.jobId },
})

Other video adapters: openaiVideo('sora-2') (pixel sizes like '1280x720', durations 4/8/12s, single input_reference image prompt part), grokVideo(...) (grok-imagine-video and grok-imagine-video-1.5 both do text-to-video + image-to-video; 1.5 adds reference-to-video — 'reference'/'character'-roled image parts → reference_images (max 7), preset voices via modelOptions.reference_audios (max 3) — 1.5-only, capped at 720p, and not combinable with a starting-frame image; only grok-imagine-video edits/extends a source video prompt part via modelOptions.mode: 'edit' | 'extend' (extend duration = added tail). Edit/extend outputs inherit the source clip's properties, so size/aspect_ratio/resolution throw in both modes and duration throws in edit mode — pass none of them there; generation uses the aspect-ratio size template like '16:9_720p' (1080p is 1.5-only), integer durations 1-15s, reports usage.billed seconds ({ quantity, unit: 'seconds' }) and exact usage.cost), byteplusVideo(...) (Seedance — aspect-ratio size template like '16:9_720p', durations 4-15s on the 2.0 family, 4-12s on 1.5-pro, 2-12s on the 1.0-pro models; reads ARK_API_KEY), openRouterVideo(...) (OpenRouter's dedicated POST /api/v1/videos gateway), and falVideo(...) (hosted models; duration typed from @fal-ai/client's EndpointTypeMap — '5' | '10' on Kling 2.6, '3'…'15' on Kling 3, '4s' | '6s' | '8s' on Veo 3.1, '5s' | '9s' on Luma; availableDurations() / snapDuration() on the curated set; see cost tracking below).

Seedance option applicability is per model and enforced server-side — Ark returns a 400 for an inapplicable field rather than ignoring it. service_tier / camera_fixed are Seedance 1.x only, frames is 1-0-pro + 1-0-pro-fast only, draft is 1-5-pro only, priority is the 2.0 family only, and duration: -1 works on 2.0 + 1-5-pro. There is no 2K tier on any model and 4k exists only on dreamina-seedance-2-0-260128. Video URLs expire 24 hours after the task completes (task record kept 7 days). Seedance is also reachable via falVideo — byteplusVideo is the direct-to-BytePlus path.

OpenRouter (@tanstack/ai-openrouter, openRouterVideo) runs the dedicated async video API (POST /api/v1/videos) and shares the same typed-duration contract — duration, size, and provider options are narrowed per model from OpenRouter's published metadata, with the same availableDurations() / snapDuration() helpers:

import { generateVideo } from '@tanstack/ai'
import { openRouterVideo } from '@tanstack/ai-openrouter'

const adapter = openRouterVideo('bytedance/seedance-2.0')
adapter.availableDurations()
// { kind: 'discrete', values: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] }
adapter.snapDuration(7.4) // 7

const sliderSeconds = 7 // raw seconds from a UI control
const { jobId } = await generateVideo({
  adapter,
  prompt: 'A timelapse of clouds',
  duration: adapter.snapDuration(sliderSeconds),
})
// Completed url is a data: URL; usage.cost carries the real billed cost.

Client hook with job tracking:

import { useGenerateVideo, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, jobId, videoStatus, isLoading } = useGenerateVideo({
  connection: fetchServerSentEvents('/api/generate/video'),
  onJobCreated: (id) => console.log('Job created:', id),
  onStatusUpdate: (status) =>
    console.log(`${status.status} (${status.progress}%)`),
})

// videoStatus: { jobId, status, progress?, url?, error?, usage? }
// result (on completion): { url }

6. Cost tracking (fal billable units)

fal bills media generation by usage-based units, not tokens. Every fal media adapter (falImage, falAudio, falSpeech, falTranscription, falVideo) surfaces the real billed quantity on the result as usage.billed ({ quantity, unit: 'units' }), read from fal's x-fal-billable-units response header — no fetch interceptor needed. It rides on the canonical TokenUsage shape (token fields are 0 for media), mirroring how duration-billed transcription reports { quantity, unit: 'seconds' }.

import { generateImage } from '@tanstack/ai'
import { falImage } from '@tanstack/ai-fal'

const result = await generateImage({
  adapter: falImage('fal-ai/flux/dev'),
  prompt: 'a serene mountain lake',
})

// usage.billed.quantity is the priced quantity. Multiply by the endpoint unit
// price (GET https://api.fal.ai/v1/models/pricing?endpoint_id=…) for exact cost.
const unitPrice = 0.025 // USD per unit, from the pricing endpoint
if (result.usage?.billed) {
  const cost = result.usage.billed.quantity * unitPrice
}

For video, the units arrive with the completed result: getVideoJobStatus() returns usage and emits a video:usage devtools event when fal reports it.

7. Durable persistence (job lifecycle + artifact bytes)

To make generations survive a server restart and be re-served later, add withGenerationPersistence from @tanstack/ai-persistence as generation middleware. It requires stores.generationRuns (a GenerationRunStore, keyed on the run's own runId, with a required threadId naming the stable slot the run fills — that is what a client hydrates by) and, when you also pass an stores.artifacts + stores.blobs pair (both or neither), it persists the generated media bytes at blob key artifacts/<runId>/<artifactId> with an ArtifactRecord per file. memoryPersistence() ships all three for dev/tests.

import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'
import {
  withGenerationPersistence,
  memoryPersistence,
  retrieveArtifact,
  retrieveBlob,
  reconstructGeneration,
} from '@tanstack/ai-persistence'

const persistence = memoryPersistence() // swap for your DB/object-store adapter

export async function POST(req: Request) {
  const { prompt, threadId } = await req.json()
  return toServerSentEventsResponse(
    generateImage({
      adapter: openaiImage('gpt-image-1'),
      prompt,
      threadId, // the slot recorded on the job + artifacts
      stream: true,
      middleware: [
        withGenerationPersistence(persistence, {
          // Stamp a durable app-origin serve URL (the GET route below) onto
          // each persisted artifact ref, and rewrite the live result's media to
          // it. Both live and restored results then render from your origin.
          artifactUrl: (ref) => `/api/artifacts?id=${ref.artifactId}`,
        }),
      ],
    }),
  )
}

// Serve the stored bytes back (GET /api/artifacts?id=…):
export async function GET(req: Request) {
  const id = new URL(req.url).searchParams.get('id') ?? ''
  const record = await retrieveArtifact(persistence, id)
  if (!record) return new Response('Not found', { status: 404 })
  const blob = await retrieveBlob(persistence, record)
  if (!blob?.body) return new Response('Not found', { status: 404 })
  return new Response(blob.body, {
    headers: { 'content-type': record.mimeType },
  })
}

That route is enough for images. Video needs Range: seeking a <video> is built on 206 / Content-Range, and Safari refuses to play a source that ignores Range at all. Resolve the header against record.size (416 when it does not fit), pass retrieveBlob(persistence, record, { range }), and answer 206 from the returned blob.range plus accept-ranges: bytes. The full route is in the persistence docs under Serve video: honour Range.

On the client, persistence is boolean only: persistence: true hydrates the last generation for the thread on mount, via the connection's hydrateGeneration handler backed by a reconstructGeneration GET route. There is no storage-adapter mode, so nothing about a generation is cached in the browser.

persistence: true requires a stable threadId, and it is a type error to set one without the other. That is the generation's scope, the slot successive runs fill (e.g. video-9-start-frame), not a link to a chat. id is the devtools label only and never a persistence key.

The hooks are transparent (like useChat): a reload repaints status / result / error, not a separate resumeSnapshot. Because the artifactUrl above stamps a durable URL onto each ref (carried on result.artifacts), the restored result rebuilds its media from those refs and serves from your own origin. Without byte storage, a reload restores status / error and result stays null.

• Building the R2/D1-backed byte stores for a Cloudflare Worker: ai-persistence/build-cloudflare-artifact-store.
• Store contracts, composePersistence, and the wiring end-to-end: docs/persistence/generation-persistence.md and the ai-core/client-persistence sub-skill.

Common Hook API

All generation hooks return the same shape:

The hook is transparent, mirroring useChat: there is no resumeSnapshot, resumeState, pendingArtifacts, or resultArtifacts field. Hooks also accept persistence: true plus a stable threadId: on mount the client hydrates the last run for that scope from the server and repaints the normal status / result / error fields, so the last run survives a reload (metadata only, never media bytes; result's media returns only with server byte storage + artifactUrl). See ai-core/client-persistence for details.

Provide either connection (streaming SSE transport) or fetcher (direct async call / server function returning Response). Use onResult to transform what is stored:

import { useGenerateSpeech, fetchServerSentEvents } from '@tanstack/ai-react'

const { result } = useGenerateSpeech({
  connection: fetchServerSentEvents('/api/generate/speech'),
  onResult: (raw) => ({
    audioUrl: `data:${raw.contentType};base64,${raw.audio}`,
    duration: raw.duration,
  }),
})
// result is typed as { audioUrl: string; duration?: number } | null

Common Mistakes

a. HIGH: Using the removed embedding() function

The embedding() function and openaiEmbed adapter were removed in v0.5.0. Agents trained on older code may still generate this pattern.

Wrong:

import { embedding } from '@tanstack/ai'
import { openaiEmbed } from '@tanstack/ai-openai'

const result = await embedding({
  adapter: openaiEmbed(),
  model: 'text-embedding-3-small',
  input: 'Hello, world!',
})

Correct -- use the provider SDK directly:

import OpenAI from 'openai'

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })

const result = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: 'Hello, world!',
})

Source: docs/migration/migration.md. Note: Fixed in v0.5.0 but agents trained on older code may still generate this pattern.

b. HIGH: Forgetting toServerSentEventsResponse with TanStack Start server functions

When using TanStack Start server functions with stream: true, you MUST wrap the stream with toServerSentEventsResponse(). Returning the raw stream from a server function will not work.

Wrong:

import { createServerFn } from '@tanstack/react-start'
import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

export const generateImageStreamFn = createServerFn({ method: 'POST' })
  .inputValidator((data: { prompt: string }) => data)
  .handler(({ data }) => {
    // BUG: returning raw stream -- client cannot parse this
    // (also a type error: an AsyncIterable is not a valid server-function return)
    return generateImage({
      adapter: openaiImage('gpt-image-1'),
      prompt: data.prompt,
      stream: true,
    })
  })

Correct:

import { createServerFn } from '@tanstack/react-start'
import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

export const generateImageStreamFn = createServerFn({ method: 'POST' })
  .inputValidator((data: { prompt: string }) => data)
  .handler(({ data }) => {
    return toServerSentEventsResponse(
      generateImage({
        adapter: openaiImage('gpt-image-1'),
        prompt: data.prompt,
        stream: true,
      }),
    )
  })

Source: maintainer interview.

c. MEDIUM: Not downloading OpenAI image URLs before they expire

OpenAI image URLs expire after 1 hour. If you store the URL and display it later, the image will silently break. Always download or display the image immediately, or convert to base64 for persistence.

import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

const result = await generateImage({
  adapter: openaiImage('dall-e-3'),
  prompt: 'A mountain landscape',
})

// GOOD: download immediately
for (const img of result.images) {
  if (img.url) {
    const response = await fetch(img.url)
    const blob = await response.blob()
    // Save blob to storage...
  }
}

// GOOD: use b64Json when available (no expiration)
// gpt-image-1 returns b64Json by default

Source: docs/media/image-generation.md.

d. MEDIUM: Using stream: true for activities that do not support streaming

Not all generation activities support streaming. Passing stream: true to an activity that does not support it may hang or produce unexpected results. Check the activity documentation before enabling streaming. All built-in activities (generateImage, generateAudio, generateSpeech, generateTranscription, generateVideo, summarize) support stream: true, but custom useGeneration setups may not.

Source: docs/media/generations.md.

e. HIGH: Passing responseMimeType or negativePrompt to Gemini Lyria

Gemini's GenerateContentConfig (used by Lyria 3 Pro / Lyria 3 Clip) does not support responseMimeType or negativePrompt. Lyria 3 Clip always returns 30-second audio/mp3; Lyria 3 Pro returns audio/mp3. These fields are not in `GeminiAudioP

…(truncated)