ai-core-media-generation

name	ai-core/media-generation
description	Image, audio, video, speech (TTS), and transcription generation using activity-specific adapters: generateImage() with openaiImage/geminiImage/byteplusImage, generateAudio() with geminiAudio/falAudio, generateVideo() with async polling (openaiVideo/geminiVideo/grokVideo/falVideo/byteplusVideo/openRouterVideo, per-model typed durations), generateSpeech() with openaiSpeech/byteplusSpeech, generateTranscription() with openaiTranscription/byteplusTranscription. React hooks: useGenerateImage, useGenerateAudio, useGenerateSpeech, useTranscription, useGenerateVideo. TanStack Start server function integration with toServerSentEventsResponse.
type	sub-skill
library	tanstack-ai
library_version	0.42.0
sources	["TanStack/ai:docs/media/generations.md","TanStack/ai:docs/media/generation-hooks.md","TanStack/ai:docs/media/image-generation.md","TanStack/ai:docs/media/audio-generation.md","TanStack/ai:docs/media/video-generation.md","TanStack/ai:docs/media/text-to-speech.md","TanStack/ai:docs/media/transcription.md","TanStack/ai:docs/advanced/debug-logging.md"]

Media Generation

Dependency note: This skill builds on ai-core. Read it first for critical rules.

All media activities (image, speech, transcription, video) follow the same server/client architecture: a generate*() function on the server, an SSE transport via toServerSentEventsResponse(), and a framework hook on the client.

Setup -- Image Generation End-to-End

Server (API route or TanStack Start server function)

// routes/api/generate/image.ts
import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

export async function POST(req: Request) {
  const { prompt, size, numberOfImages } = await req.json()

  const stream = generateImage({
    adapter: openaiImage('gpt-image-1'),
    prompt,
    size,
    numberOfImages,
    stream: true,
  })

  return toServerSentEventsResponse(stream)
}

Client (React)

import { useGenerateImage, fetchServerSentEvents } from '@tanstack/ai-react'
import { useState } from 'react'

function ImageGenerator() {
  const [prompt, setPrompt] = useState('')
  const { generate, result, isLoading, error, reset } = useGenerateImage({
    connection: fetchServerSentEvents('/api/generate/image'),
  })

  return (
    <div>
      <input
        value={prompt}
        onChange={(e) => setPrompt(e.target.value)}
        placeholder="Describe an image..."
      />
      <button
        onClick={() => generate({ prompt })}
        disabled={isLoading || !prompt.trim()}
      >
        {isLoading ? 'Generating...' : 'Generate'}
      </button>

      {error && <p>Error: {error.message}</p>}

      {result?.images.map((img, i) => (
        <img
          key={i}
          src={img.url || `data:image/png;base64,${img.b64Json}`}
          alt={img.revisedPrompt || 'Generated '}
        />
      ))}

      {result && Clear}
    
  )
}

TanStack Start: Server Function Streaming (recommended)

When using TanStack Start, return toServerSentEventsResponse() from a server function. The client fetcher receives a Response and the hook parses it as SSE automatically:

// lib/server-functions.ts
import { createServerFn } from '@tanstack/react-start'
import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

export const generateImageStreamFn = createServerFn({ method: 'POST' })
  .inputValidator((data: { prompt: string; model?: string }) => data)
  .handler(({ data }) => {
    return toServerSentEventsResponse(
      generateImage({
        adapter: openaiImage(data.model ?? 'gpt-image-1'),
        prompt: data.prompt,
        stream: true,
      }),
    )
  })

import { useGenerateImage } from '@tanstack/ai-react'
import { generateImageStreamFn } from '../lib/server-functions'

function ImageGenerator() {
  const { generate, result, isLoading } = useGenerateImage({
    fetcher: (input) => generateImageStreamFn({ data: input }),
  })

  return (
    <button
      onClick={() => generate({ prompt: 'A sunset over mountains' })}
      disabled={isLoading}
    >
      {isLoading ? 'Generating...' : 'Generate'}
    </button>
  )
}

---

Core Patterns

1. Image Generation

Supported adapters: openaiImage (dall-e-2, dall-e-3, gpt-image-1, gpt-image-1-mini, gpt-image-2), geminiImage (gemini-3.1-flash-image, gemini-3.1-flash-lite-image, gemini-3-pro-image, imagen-4.0-generate-001, etc.) and byteplusImage (Seedream — seedream-4-0-250828, seedream-4-5-251128, the 5.0 family).

Use the GA Gemini image ids. gemini-3.1-flash-image-preview and gemini-3-pro-image-preview were shut down on 2026-06-25 and now 404. They survive in the type union only as deprecated aliases so existing code keeps compiling — a call to them typechecks and then fails at runtime. Use gemini-3.1-flash-image / gemini-3-pro-image instead.

Seedream quirks: watermark defaults to true (pass modelOptions: { watermark: false } for a clean image), size is a token ('1K' | '2K' | '4K') or explicit '2048x2048' pixels but never a mix, and numberOfImages is an upper bound — Seedream has no n, so it maps onto group-image mode and the model may return fewer. Reads ARK_API_KEY.

import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'
import { geminiImage } from '@tanstack/ai-gemini'

// OpenAI with quality/background options
const openaiResult = await generateImage({
  adapter: openaiImage('gpt-image-1'),
  prompt: 'A cat wearing a hat',
  size: '1024x1024',
  numberOfImages: 2,
  modelOptions: {
    quality: 'high',
    background: 'transparent',
    outputFormat: 'png',
  },
})

// Gemini native model with aspect-ratio sizes
const geminiResult = await generateImage({
  adapter: geminiImage('gemini-3.1-flash-image'),
  prompt: 'A futuristic cityscape at night',
  size: '16:9_4K',
})

// Gemini Imagen model
const imagenResult = await generateImage({
  adapter: geminiImage('imagen-4.0-generate-001'),
  prompt: 'A landscape photo',
  : { :  },
})

Result shape: ImageGenerationResult with images array where each entry has b64Json?, url?, and revisedPrompt?. OpenAI image URLs expire after 1 hour -- download or display immediately.

Image-conditioned generation: multimodal prompt parts

Both generateImage() and generateVideo() accept the prompt either as a plain string or as an ordered array of content parts (TextPart / ImagePart / VideoPart / AudioPart — the same shapes used elsewhere in TanStack AI). Part order is meaningful: natively multimodal providers (Gemini, OpenRouter) receive parts in order; named-field providers (OpenAI, fal, xAI) extract media parts and flatten the text. Prompt text is always sent verbatim — to reference inputs from the prompt, write the provider's own syntax (fal @Image1, OpenAI "image 1" prose); the SDK never injects or rewrites markers. Each media part may carry an optional metadata.role hint that adapters use to route the part to the provider-specific field. The accepted part types are narrowed per model at compile time via the adapter's input-modality map.

import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

// Image-to-image (OpenAI gpt-image-2 / gpt-image-1, dall-e-2)
await generateImage({
  adapter: openaiImage('gpt-image-2'),
  prompt: [
    { type: 'text', content: 'Turn this into a cinematic product photo' },
    { type: 'image', source: { type: 'url', value: 'https://…/product.png' } },
  ],
})

// Multi-reference (up to 16 for gpt-image models; up to ~14 for Gemini native
// — a provider limit, not enforced by the SDK)
await generateImage({
  adapter: openaiImage('gpt-image-2'),
  prompt: [
    { type: 'text', content: 'Apply the second image as style to the first' },
    { type: 'image', source: { type: 'url', value: 'https://…/product.png' } },
    { type: 'image', source: { type: 'url', value: 'https://…/style.png' } },
  ],
})


 ({
  : (),
  : [
    { : , :  },
    { : , : { : , : photoUrl } },
    {
      : ,
      : { : , : maskUrl },
      : { :  },
    },
  ],
})



 { generateVideo }  
 { falVideo }  

 ({
  : (),
  : [
    { : , : { : , : firstFrameUrl } },
    { : , :  },
    {
      : ,
      : { : , : lastFrameUrl },
      : { :  },
    },
  ],
})

URL inputs that require an upload throw by default. Most adapters pass a type: 'url' source straight through to the provider. Three paths can't — OpenAI images.edit(), OpenAI Sora input_reference, and Gemini Veo — because the provider only accepts uploaded bytes (Veo also takes a gs:// reference). For those, an HTTP(S) URL would have to be downloaded and buffered in memory, which can OOM constrained runtimes, so they throw on an HTTP(S) URL image input by default. Pass a data: URI (or gs:// for Veo), or opt in with allowUrlFetch: true on the adapter config (createOpenaiImage(model, apiKey, { allowUrlFetch: true }), and likewise on createOpenaiVideo / createGeminiVideo). data: URIs never need the flag.

Role hints (metadata.role):

Role	Maps to
'reference'	fal reference_image_urls; OpenRouter video input_references[]; Gemini multimodal part; positional otherwise
'character'	Same as 'reference'; Veo referenceImages; OpenRouter input_references[]
'mask'	OpenAI mask (gpt-image-2, gpt-image-1, dall-e-2); fal mask_url
'control'	fal control_image_url (ControlNet / depth / pose)
'start_frame'	fal start_image_url (or the endpoint's field, e.g. image_url on Kling i2v); OpenRouter frame_images[] first_frame; Veo image
'end_frame'	fal end_image_url (or e.g. tail_image_url / last_frame_url); OpenRouter frame_images[] last_frame; Veo lastFrame

Provider support matrix:

Provider	generateImage image parts	generateVideo image parts
OpenAI	gpt-image-2 / gpt-image-1 / -mini → images.edit() (up to 16). dall-e-2 → edit (1). dall-e-3 throws.	Sora-2 / -pro → input_reference (single). Throws if >1.
Gemini	Native (gemini-*-flash-image, "nano-banana") → multimodal contents. Imagen throws.	Veo → first un-roled / 'start_frame' image is the input image; 'end_frame' → lastFrame; 'reference' / 'character' → referenceImages. Omni Flash sends image/video parts as interaction content blocks (no role routing).
fal	Per-endpoint field names from a generated map (pnpm generate:fal-image-fields). Defaults: 1 input → image_url; >1 → image_urls; roles → mask_url / control_image_url / reference_image_urls.	Per-endpoint map (e.g. Kling i2v start frame → image_url). Defaults: 1 input → image_url; start_frame/end_frame → start_image_url/end_image_url; reference → reference_image_urls.
Grok	grok-imagine models → /v1/images/edits JSON endpoint (≤3 sources, addressed by xAI in request order; prompt sent verbatim; mask/control throw). grok-2-image-1212 throws.	Un-roled / 'start_frame' image → starting frame; 'reference' / 'character' → reference_images (1.5). Starting frame and reference inputs cannot be combined. A video part + modelOptions.mode: 'edit' | 'extend' routes to /videos/edits / /videos/extensions on grok-imagine-video only.
OpenRouter	Prompt parts map 1:1 onto multimodal text / image_url content parts, preserving interleaved order.

Video and audio prompt parts follow the same metadata.role convention for video-to-video and lipsync flows on fal. Grok accepts one source video part on grok-imagine-video with modelOptions.mode: 'edit' | 'extend' and rejects audio parts. Other providers throw when those parts are passed.

2. Audio Generation (Music, Sound Effects)

Distinct from TTS — generateAudio() produces non-speech audio content. Supported adapters: geminiAudio (Lyria 3 Pro / Lyria 3 Clip) and falAudio (MiniMax Music, DiffRhythm, Stable Audio, ElevenLabs SFX, etc.).

import { generateAudio } from '@tanstack/ai'
import { falAudio } from '@tanstack/ai-fal'

const result = await generateAudio({
  adapter: falAudio('fal-ai/diffrhythm'),
  prompt: 'An upbeat electronic track with synths',
  duration: 10,
})

// result.audio.url or result.audio.b64Json (provider-dependent)
// result.audio.contentType e.g. "audio/mpeg"

Client hook:

import { useGenerateAudio, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, isLoading } = useGenerateAudio({
  connection: fetchServerSentEvents('/api/generate/audio'),
})

// Trigger: generate({ prompt: 'Upbeat synths', duration: 10 })
// Play:    <audio src={result.audio.url} controls />

3. Text-to-Speech

Adapters: openaiSpeech (tts-1, tts-1-hd, gpt-4o-audio-preview) and byteplusSpeech (seed-audio-1.0).

BytePlus Seed Speech is a separate product from ModelArk — it reads BYTEPLUS_VOICE_API_KEY, not ARK_API_KEY, and an Ark key there fails with 45000010 Invalid X-Api-Key. Output is capped at 120 seconds. There is no top-level speaker field — voice is sent as references: [{ speaker }], and modelOptions.references replaces that array rather than merging, so passing references for voice cloning silently drops voice. Voice ids ending _uranus_bigtts are TTS 2.0, _mars_bigtts / _moon_bigtts are TTS 1.0, and *_emo_v2_* are the 1.0 voices that accept emotion tags. Formats: wav, mp3, pcm, ogg_opus; watermark is also available on modelOptions.

import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

const result = await generateSpeech({
  adapter: openaiSpeech('tts-1-hd'),
  text: 'Hello, welcome to TanStack AI!',
  voice: 'alloy', // alloy | echo | fable | onyx | nova | shimmer | ash | ballad | coral | sage | verse
  format: 'mp3', // mp3 | opus | aac | flac | wav | pcm
  speed: 1.0, // 0.25 to 4.0
})

// result.audio is base64-encoded audio
// result.format is the output format string
// result.contentType is the MIME type (e.g. "audio/mpeg")

Client hook:

import { useGenerateSpeech, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, isLoading } = useGenerateSpeech({
  connection: fetchServerSentEvents('/api/generate/speech'),
})

// Trigger: generate({ text: 'Hello!', voice: 'alloy' })
// Play:   <audio src={`data:audio/${result.format};base64,${result.audio}`} controls />

4. Audio Transcription

Adapters: openaiTranscription (whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize) and byteplusTranscription (seed-asr — synchronous, no polling; audio up to 2 hours / 100 MB; also reads BYTEPLUS_VOICE_API_KEY).

Capturing audio in the browser: Use useAudioRecorder from @tanstack/ai-react to record directly in the browser, then pass the recording as the audio input to generate(), or use recording.part as a prompt part in chat/generation calls. No transcoding or extra dependencies required — the recorder returns the native browser format (audio/webm or audio/mp4). For transcription, wrap it as a data: URL so the provider gets the real content type; passing raw recording.base64 makes the adapter assume audio/mpeg and mislabel the webm/mp4 bytes.

const { isRecording, start, stop } = useAudioRecorder()
const { generate } = useTranscription({
  connection: fetchServerSentEvents('/api/transcribe'),
})
// ...
const recording = await stop()
const mimeType = recording.mimeType.split(';')[0] // strip ;codecs=...
await generate({ audio: `data:${mimeType};base64,${recording.base64}` })

import { generateTranscription } from '@tanstack/ai'
import { openaiTranscription } from '@tanstack/ai-openai'

const result = await generateTranscription({
  adapter: openaiTranscription('whisper-1'),
  audio: audioFile, // File, Blob, base64 string, or data URL
  language: 'en',
  responseFormat: 'verbose_json',
  modelOptions: {
    timestamp_granularities: ['word', 'segment'],
  },
})

// result.text       -- full transcribed text
// result.language   -- detected/specified language
// result.duration   -- audio duration in seconds
// result.segments   -- timestamped segments (word-level timestamps are in result.words)

For speaker diarization, use openaiTranscription('gpt-4o-transcribe-diarize'). When no response format is given it defaults the request to response_format: 'diarized_json' and chunking_strategy: 'auto' (a top-level responseFormat of 'json'/'text' opts out of speaker segments); do not pass prompt, include, or timestamp_granularities with this model.

Client hook:

import { useTranscription, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, isLoading } = useTranscription({
  connection: fetchServerSentEvents('/api/transcribe'),
})

// Trigger: generate({ audio: dataUrl, language: 'en' })

5. Video Generation (Experimental -- async polling)

Video generation uses a jobs/polling architecture. The server creates a job, polls for status, and streams updates to the client. Adapters: openaiVideo (Sora), geminiVideo (Veo / Omni Flash), grokVideo, byteplusVideo (Seedance), falVideo (Kling, MiniMax, Hunyuan, …), and openRouterVideo (OpenRouter's dedicated POST /api/v1/videos gateway — Seedance, Veo, Wan, Kling, Sora 2 Pro and others through one API key; getVideoJobStatus() returns the video as a data: URL since OpenRouter's download URLs require the API key, and surfaces the gateway-reported cost as usage.cost).

import {
  generateVideo,
  getVideoJobStatus,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { openaiVideo } from '@tanstack/ai-openai'

// Non-streaming: manual polling loop
const { jobId } = await generateVideo({
  adapter: openaiVideo('sora-2'),
  prompt: 'A golden retriever playing in sunflowers',
  size: '1280x720',
  duration: 8,
})

let status = await getVideoJobStatus({ adapter: openaiVideo('sora-2'), jobId })
while (status.status !== 'completed' && status.status !== 'failed') {
  await new Promise((r) => setTimeout(r, 5000))
  status = await getVideoJobStatus({ adapter: openaiVideo('sora-2'), jobId })
}

// Streaming: server handles polling, client gets real-time updates
const stream = generateVideo({
  adapter: openaiVideo('sora-2'),
  prompt: ,
  : ,
  : ,
  : ,
})
 (stream)

Google Veo (@tanstack/ai-gemini) uses the same jobs/polling flow. Its duration option is typed per model (4 | 6 | 8 for the Veo 3.1 models); use adapter.snapDuration(seconds) to coerce raw seconds and adapter.availableDurations() to enumerate the valid set. Image prompt parts route by metadata.role: first un-roled / 'start_frame' image → input image, 'end_frame' → lastFrame, 'reference' / 'character' → referenceImages:

import { geminiVideo } from '@tanstack/ai-gemini'

const adapter = geminiVideo('veo-3.1-generate-preview')
adapter.availableDurations() // { kind: 'discrete', values: [4, 6, 8] }

const { jobId } = await generateVideo({
  adapter,
  prompt: 'A golden retriever playing in sunflowers',
  size: '16:9', // Veo sizes are aspect ratios: '16:9' | '9:16'
  duration: adapter.snapDuration(7), // 6
  modelOptions: { resolution: '1080p', generateAudio: true },
})
// Note: Veo result URLs require the Google API key to download
// (x-goog-api-key header or ?key= query parameter).

Gemini Omni Flash (geminiVideo('gemini-omni-flash-preview')) is served by the Interactions API instead of Veo's operations flow — same adapter, routed by model. Clips are 720p; duration is any number of seconds in the 3–10 range (fractional ok, default 10 — availableDurations() reports the range), size is the aspect ratio ('16:9' | '9:16'), and the finished video arrives inline as a data:video/mp4;base64,… URL (no key needed to use it). Image/video prompt parts are sent as interaction content blocks, grouped as images, then videos, then text (no metadata.role routing); data sources go inline, url sources pass through as-is (never downloaded — use Gemini Files API URIs for remote media). For conversational editing, pass a prior generation's jobId as modelOptions.previous_interaction_id with a prompt describing the change:

import { geminiVideo } from '@tanstack/ai-gemini'

const omni = geminiVideo('gemini-omni-flash-preview')
const first = await generateVideo({
  adapter: omni,
  prompt: 'A violinist outdoors',
})
// …poll first.jobId to completion, then edit it:
const edited = await generateVideo({
  adapter: omni,
  prompt: 'Make the violin invisible',
  modelOptions: { previous_interaction_id: first.jobId },
})

Other video adapters: openaiVideo('sora-2') (pixel sizes like '1280x720', durations 4/8/12s, single input_reference image prompt part), grokVideo(...) (grok-imagine-video and grok-imagine-video-1.5 both do text-to-video + image-to-video; 1.5 adds reference-to-video — 'reference'/'character'-roled image parts → reference_images (max 7), preset voices via modelOptions.reference_audios (max 3) — 1.5-only, capped at 720p, and not combinable with a starting-frame image; only grok-imagine-video edits/extends a source video prompt part via modelOptions.mode: 'edit' | 'extend' (extend duration = added tail). Edit/extend outputs inherit the source clip's properties, so size/aspect_ratio/resolution throw in both modes and duration throws in edit mode — pass none of them there; generation uses the aspect-ratio size template like '16:9_720p' (1080p is 1.5-only), integer durations 1-15s, reports usage.billed seconds ({ quantity, unit: 'seconds' }) and exact usage.cost), byteplusVideo(...) (Seedance — aspect-ratio size template like '16:9_720p', durations 4-15s on the 2.0 family, 4-12s on 1.5-pro, 2-12s on the 1.0-pro models; reads ARK_API_KEY), openRouterVideo(...) (OpenRouter's dedicated POST /api/v1/videos gateway), and falVideo(...) (hosted models; duration typed from @fal-ai/client's EndpointTypeMap — '5' | '10' on Kling 2.6, '3'…'15' on Kling 3, '4s' | '6s' | '8s' on Veo 3.1, '5s' | '9s' on Luma; availableDurations() / snapDuration() on the curated set; see cost tracking below).

Seedance option applicability is per model and enforced server-side — Ark returns a 400 for an inapplicable field rather than ignoring it. service_tier / camera_fixed are Seedance 1.x only, frames is 1-0-pro + 1-0-pro-fast only, draft is 1-5-pro only, priority is the 2.0 family only, and duration: -1 works on 2.0 + 1-5-pro. There is no 2K tier on any model and 4k exists only on dreamina-seedance-2-0-260128. Video URLs expire 24 hours after the task completes (task record kept 7 days). Seedance is also reachable via falVideo — byteplusVideo is the direct-to-BytePlus path.

OpenRouter (@tanstack/ai-openrouter, openRouterVideo) runs the dedicated async video API (POST /api/v1/videos) and shares the same typed-duration contract — duration, size, and provider options are narrowed per model from OpenRouter's published metadata, with the same availableDurations() / snapDuration() helpers:

import { openRouterVideo } from '@tanstack/ai-openrouter'

const adapter = openRouterVideo('bytedance/seedance-2.0')
adapter.availableDurations()
// { kind: 'discrete', values: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] }
adapter.snapDuration(7.4) // 7

const sliderSeconds = 7 // raw seconds from a UI control
const { jobId } = await generateVideo({
  adapter,
  prompt: 'A timelapse of clouds',
  duration: adapter.snapDuration(sliderSeconds),
})
// Completed url is a data: URL; usage.cost carries the real billed cost.

Client hook with job tracking:

import { useGenerateVideo, fetchServerSentEvents } from '@tanstack/ai-react'

const { generate, result, jobId, videoStatus, isLoading } = useGenerateVideo({
  connection: fetchServerSentEvents('/api/generate/video'),
  onJobCreated: (id) => console.log('Job created:', id),
  onStatusUpdate: (status) =>
    console.log(`${status.status} (${status.progress}%)`),
})

// videoStatus: { jobId, status, progress?, url?, error?, usage? }
// result (on completion): { url }

6. Cost tracking (fal billable units)

fal bills media generation by usage-based units, not tokens. Every fal media adapter (falImage, falAudio, falSpeech, falTranscription, falVideo) surfaces the real billed quantity on the result as usage.billed ({ quantity, unit: 'units' }), read from fal's x-fal-billable-units response header — no fetch interceptor needed. It rides on the canonical TokenUsage shape (token fields are 0 for media), mirroring how duration-billed transcription reports { quantity, unit: 'seconds' }.

import { generateImage } from '@tanstack/ai'
import { falImage } from '@tanstack/ai-fal'

const result = await generateImage({
  adapter: falImage('fal-ai/flux/dev'),
  prompt: 'a serene mountain lake',
})

// usage.billed.quantity is the priced quantity. Multiply by the endpoint unit
// price (GET https://api.fal.ai/v1/models/pricing?endpoint_id=…) for exact cost.
if (result.usage?.billed) {
  const cost = result.usage.billed.quantity * unitPrice
}

For video, the units arrive with the completed result: getVideoJobStatus() returns usage and emits a video:usage devtools event when fal reports it.

7. Durable persistence (job lifecycle + artifact bytes)

To make generations survive a server restart and be re-served later, add withGenerationPersistence from @tanstack/ai-persistence as generation middleware. It requires stores.generationRuns (a GenerationRunStore, keyed on the run's own runId, with a required threadId naming the stable slot the run fills — that is what a client hydrates by) and, when you also pass an stores.artifacts + stores.blobs pair (both or neither), it persists the generated media bytes at blob key artifacts/<runId>/<artifactId> with an ArtifactRecord per file. memoryPersistence() ships all three for dev/tests.

import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'
import {
  withGenerationPersistence,
  memoryPersistence,
  retrieveArtifact,
  retrieveBlob,
  reconstructGeneration,
} from '@tanstack/ai-persistence'

const persistence = memoryPersistence() // swap for your DB/object-store adapter

export async function POST(req: Request) {
  const { prompt, threadId } = await req.json()
  return toServerSentEventsResponse(
    generateImage({
      adapter: openaiImage('gpt-image-1'),
      prompt,
      threadId, // the slot recorded on the job + artifacts
      stream: true,
      middleware: [
        withGenerationPersistence(persistence, {
          // Stamp a durable app-origin serve URL (the GET route below) onto
          // each persisted artifact ref, and rewrite the live result's media to
          // it. Both live and restored results then render from your origin.
          artifactUrl: (ref) => `/api/artifacts?id=${ref.artifactId}`,
        }),
      ],
    }),
  )
}

// Serve the stored bytes back (GET /api/artifacts?id=…):
   () {
   id =  (req.)..() ?? 
   record =  (persistence, id)
   (!record)   (, { :  })
   blob =  (persistence, record)
   (!blob?.)   (, { :  })
    (blob., {
    : { : record. },
  })
}

That route is enough for images. Video needs Range: seeking a <video> is built on 206 / Content-Range, and Safari refuses to play a source that ignores Range at all. Resolve the header against record.size (416 when it does not fit), pass retrieveBlob(persistence, record, { range }), and answer 206 from the returned blob.range plus accept-ranges: bytes. The full route is in the persistence docs under Serve video: honour Range.

On the client, persistence is boolean only: persistence: true hydrates the last generation for the thread on mount, via the connection's hydrateGeneration handler backed by a reconstructGeneration GET route. There is no storage-adapter mode, so nothing about a generation is cached in the browser.

persistence: true requires a stable threadId, and it is a type error to set one without the other. That is the generation's scope, the slot successive runs fill (e.g. video-9-start-frame), not a link to a chat. id is the devtools label only and never a persistence key.

The hooks are transparent (like useChat): a reload repaints status / result / error, not a separate resumeSnapshot. Because the artifactUrl above stamps a durable URL onto each ref (carried on result.artifacts), the restored result rebuilds its media from those refs and serves from your own origin. Without byte storage, a reload restores status / error and result stays null.

• Building the R2/D1-backed byte stores for a Cloudflare Worker: ai-persistence/build-cloudflare-artifact-store.
• Store contracts, composePersistence, and the wiring end-to-end: docs/persistence/generation-persistence.md and the ai-core/client-persistence sub-skill.

---

Common Hook API

All generation hooks return the same shape:

Property	Type	Description
generate	(input) => Promise<void>	Trigger generation
result	T | null	Result (optionally transformed via onResult)
isLoading	boolean	Whether generation is in progress
error	Error | undefined	Current error
status	GenerationClientState	'idle' | 'generating' | 'success' | 'error'
stop	() => void	Abort current generation
reset	() => void	Clear state and the in-memory snapshot
runId	string | null	Id of the job WHILE it runs; null when idle

The hook is transparent, mirroring useChat: there is no resumeSnapshot, resumeState, pendingArtifacts, or resultArtifacts field. Hooks also accept persistence: true plus a stable threadId: on mount the client hydrates the last run for that scope from the server and repaints the normal status / result / error fields, so the last run survives a reload (metadata only, never media bytes; result's media returns only with server byte storage + artifactUrl). See ai-core/client-persistence for details.

Provide either connection (streaming SSE transport) or fetcher (direct async call / server function returning Response). Use onResult to transform what is stored:

const { result } = useGenerateSpeech({
  connection: fetchServerSentEvents('/api/generate/speech'),
  onResult: (raw) => ({
    audioUrl: `data:${raw.contentType};base64,${raw.audio}`,
    duration: raw.duration,
  }),
})
// result is typed as { audioUrl: string; duration?: number } | null

---

Common Mistakes

a. HIGH: Using the removed embedding() function

The embedding() function and openaiEmbed adapter were removed in v0.5.0. Agents trained on older code may still generate this pattern.

Wrong:

import { embedding } from '@tanstack/ai'
import { openaiEmbed } from '@tanstack/ai-openai'

const result = await embedding({
  adapter: openaiEmbed(),
  model: 'text-embedding-3-small',
  input: 'Hello, world!',
})

Correct -- use the provider SDK directly:

import OpenAI from 'openai'

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })

const result = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: 'Hello, world!',
})

Source: docs/migration/migration.md. Note: Fixed in v0.5.0 but agents trained on older code may still generate this pattern.

b. HIGH: Forgetting toServerSentEventsResponse with TanStack Start server functions

When using TanStack Start server functions with stream: true, you MUST wrap the stream with toServerSentEventsResponse(). Returning the raw stream from a server function will not work.

Wrong:

export const generateImageStreamFn = createServerFn({ method: 'POST' }).handler(
  ({ data }) => {
    // BUG: returning raw stream -- client cannot parse this
    return generateImage({
      adapter: openaiImage('gpt-image-1'),
      prompt: data.prompt,
      stream: true,
    })
  },
)

Correct:

import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

export const generateImageStreamFn = createServerFn({ method: 'POST' }).handler(
  ({ data }) => {
    return toServerSentEventsResponse(
      generateImage({
        adapter: openaiImage('gpt-image-1'),
        prompt: data.prompt,
        stream: true,
      }),
    )
  },
)

Source: maintainer interview.

c. MEDIUM: Not downloading OpenAI image URLs before they expire

OpenAI image URLs expire after 1 hour. If you store the URL and display it later, the image will silently break. Always download or display the image immediately, or convert to base64 for persistence.

const result = await generateImage({
  adapter: openaiImage('dall-e-3'),
  prompt: 'A mountain landscape',
})

// GOOD: download immediately
for (const img of result.images) {
  if (img.url) {
    const response = await fetch(img.url)
    const blob = await response.blob()
    // Save blob to storage...
  }
}

// GOOD: use b64Json when available (no expiration)
// gpt-image-1 returns b64Json by default

Source: docs/media/image-generation.md.

d. MEDIUM: Using stream: true for activities that do not support streaming

Not all generation activities support streaming. Passing stream: true to an activity that does not support it may hang or produce unexpected results. Check the activity documentation before enabling streaming. All built-in activities (generateImage, generateAudio, generateSpeech, generateTranscription, generateVideo, summarize) support stream: true, but custom useGeneration setups may not.

Source: docs/media/generations.md.

e. HIGH: Passing responseMimeType or negativePrompt to Gemini Lyria

Gemini's GenerateContentConfig (used by Lyria 3 Pro / Lyria 3 Clip) does not support responseMimeType or negativePrompt. Lyria 3 Clip always returns 30-second audio/mp3; Lyria 3 Pro returns audio/mp3. These fields are not in GeminiAudioProviderOptions — don't reach for them via as any.

// WRONG — both fields are silently ignored or rejected by the SDK
generateAudio({
  adapter: geminiAudio('lyria-3-pro-preview'),
  prompt: 'ambient piano',
  modelOptions: {
    responseMimeType: 'audio/wav', // unsupported
    negativePrompt: 'vocals', // unsupported
  } as any,
})

// CORRECT — shape the prompt itself for what you want
generateAudio({
  adapter: geminiAudio('lyria-3-pro-preview'),
  prompt: 'ambient piano, no vocals',
})

Source: Gemini API GenerateContentConfig type; docs/media/audio-generation.md.

f. MEDIUM: Passing duration to Lyria expecting it to control length

Lyria 3 Clip is fixed at 30 seconds — the duration option is ignored on that model. Lyria 3 Pro accepts duration via natural-language in the prompt ("2-minute ambient track with a 30-second build"), not via the duration field. duration works for fal audio models (mapped to each model's native field like music_length_ms or seconds_total), but not for Lyria.

// For Lyria: put length guidance in the prompt
generateAudio({
  adapter: geminiAudio('lyria-3-pro-preview'),
  prompt: 'A 2-minute ambient piano piece with gentle strings',
  // duration: 120  // ← does nothing; rely on the prompt
})

// For fal: duration works and is translated per-model
generateAudio({
  adapter: falAudio('fal-ai/minimax-music/v2'),
  prompt: 'upbeat synth melody',
  duration: 60, // → music_length_ms: 60_000
})

Source: Google Lyria 3 docs; docs/media/audio-generation.md.

g. MEDIUM: Gemini TTS multi-speaker with 0 or 3+ speakers

multiSpeakerVoiceConfig.speakerVoiceConfigs is validated to be length 1 or 2. Passing an empty array or three+ entries throws at the adapter boundary (not at Gemini's API) with a clear error. Don't try to work around it with as any.

generateSpeech({
  adapter: geminiSpeech('gemini-2.5-pro-preview-tts'),
  text: '[Alice] Hi. [Bob] Hello!',
  modelOptions: {
    multiSpeakerVoiceConfig: {
      speakerVoiceConfigs: [
        {
          speaker: 'Alice',
          voiceConfig: { prebuiltVoiceConfig: { voiceName: 'Kore' } },
        },
        {
          speaker: 'Bob',
          voiceConfig: { prebuiltVoiceConfig: { voiceName: 'Puck' } },
        },
      ],
    },
  },
})

Source: Gemini TTS adapter validation; CodeRabbit review of PR #463.

h. HIGH: Passing image prompt parts to a model that doesn't support image-conditioned generation

Not every model accepts image-conditioned prompts. The prompt type is narrowed per model, so passing an image part to a text-only model (dall-e-3, Imagen, grok-2-image) is a compile-time error; adapters also throw a clear runtime error as a backstop, so users learn at call time rather than getting silently wrong output.

// WRONG — dall-e-3 has no edit/inputs API; image parts are a type error
generateImage({
  adapter: openaiImage('dall-e-3'),
  prompt: [
    { type: 'text', content: 'Edit this' },
    { type: 'image', source: { type: 'url', value: url } }, // ❌ type error
  ],
})

// WRONG — Imagen is text-to-image only; same compile-time rejection
generateImage({
  adapter: geminiImage('imagen-4.0-generate-001'),
  prompt: [
    { type: 'text', content: 'Edit this' },
    { type: 'image', source: { type: 'url', value: url } }, // ❌ type error
  ],
})

// CORRECT — use a model that supports image-conditioned generation
generateImage({
  adapter: openaiImage('gpt-image-2'), // edits up to 16 images
  prompt: [
    { type: 'text', content: 'Edit this' },
    { type: 'image', source: { type: 'url', : url } },
  ],
})

({
  : (), 
  : [
    { : , :  },
    { : , : { : , : url } },
  ],
})

Source: docs/media/image-generation.md, docs/media/video-generation.md.

i. LOW: Writing a logging middleware to see media chunks flow through

Every media activity — generateAudio, generateSpeech, generateTranscription, generateImage, generateVideo — accepts the same debug?: DebugOption option that chat() does. Reach for debug instead of wiring up logging middleware.

// When a speech generation sounds wrong or a transcription returns garbage
generateSpeech({
  adapter: openaiSpeech('tts-1'),
  text: 'Hello',
  debug: { provider: true, output: true }, // raw SDK chunks + yielded chunks
})

See the ai-core/debug-logging sub-skill for full details on categories and piping into a custom logger.

Source: docs/advanced/debug-logging.md.

---

Cross-References

• See also: ai-core/adapter-configuration/SKILL.md -- Each media activity requires a specific activity adapter (e.g., openaiImage for images, openaiSpeech for speech, openaiTranscription for transcription, openaiVideo for video). The adapter-configuration skill covers provider setup, API keys, and model selection.
• See also: ai-core/debug-logging/SKILL.md -- When a media request returns unexpected output or fails mid-stream, toggle debug: true on any generate*() call to see request metadata, raw provider chunks, and errors. Covers per-category toggling and piping into pino/winston.