ai-core-media-generation

# Media Generation > **Dependency note:** This skill builds on ai-core. Read it first for critical rules. All media activities (image, speech, transcription, video) follow the same server/client architecture: a `generate*()` function on the server, an SSE transport via `toServerSentEventsResponse()`, and a framework hook on the client. ## Setup -- Image Generation End-to-End ### Server (API route or TanStack Start server function) ```typescript // routes/api/generate/image.ts import { generateImage, toServerSentEventsResponse } from '@tanstack/ai' import { openaiImage } from '@tanstack/ai-openai' export async function POST(req: Request) { const { prompt, size, numberOfImages } = await req.json() const stream = generateImage({ adapter: openaiImage('gpt-image-1'), prompt, size, numberOfImages, stream: true, }) return toServerSentEventsResponse(stream) } ``` ### Client (React) ```tsx import { useGenerateImage, fetchServerSentEvents } from '@tanstack/ai-react' import { useState } from 'react' function ImageGenerator() { const [prompt, setPrompt] = useState('') const { generate, result, isLoading, error, reset } = useGenerateImage({ connection: fetchServerSentEvents('/api/generate/image'), }) return ( <div> <input value={prompt} onChange={(e) => setPrompt(e.target.value)} placeholder="Describe an image..." /> <button onClick={() => generate({ prompt })} disabled={isLoading || !prompt.trim()} > {isLoading ? 'Generating...' : 'Generate'} </button> {error && <p>Error: {error.message}</p>} {result?.images.map((img, i) => ( <img key={i} src={img.url || `data:image/png;base64,${img.b64Json}`} alt={img.revisedPrompt || 'Generated image'} /> ))} {result && <button onClick={reset}>Clear</button>} </div> ) } ``` ### TanStack Start: Server Function Streaming (recommended) When using TanStack Start, return `toServerSentEventsResponse()` from a server function. The client fetcher receives a `Response` and the hook parses it as SSE automatically: ```typescript // lib/server-functions.ts import { createServerFn } from '@tanstack/react-start' import { generateImage, toServerSentEventsResponse } from '@tanstack/ai' import { openaiImage } from '@tanstack/ai-openai' import type { OpenAIImageModel } from '@tanstack/ai-openai' export const generateImageStreamFn = createServerFn({ method: 'POST' }) .inputValidator((data: { prompt: string; model?: OpenAIImageModel }) => data) .handler(({ data }) => { return toServerSentEventsResponse( generateImage({ adapter: openaiImage(data.model ?? 'gpt-image-1'), prompt: data.prompt, stream: true, }), ) }) ``` ```tsx import { useGenerateImage } from '@tanstack/ai-react' import { generateImageStreamFn } from '../lib/server-functions' function ImageGenerator() { const { generate, result, isLoading } = useGenerateImage({ fetcher: (input) => generateImageStreamFn({ data: input }), }) return ( <button onClick={() => generate({ prompt: 'A sunset over mountains' })} disabled={isLoading} > {isLoading ? 'Generating...' : 'Generate'} </button> ) } ``` --- ## Core Patterns ### 1. Image Generation Supported adapters: `openaiImage` (dall-e-2, dall-e-3, gpt-image-1, gpt-image-1-mini, gpt-image-2), `geminiImage` (gemini-3.1-flash-image, gemini-3.1-flash-lite-image, gemini-nano-banana-2.1, gemini-3-pro-image, imagen-4.0-generate-001, etc.) and `byteplusImage` (Seedream — `seedream-4-0-250828`, `seedream-4-5-251128`, the 5.0 family). > **Use the GA Gemini image ids.** `gemini-3.1-flash-image-preview` and > `gemini-3-pro-image-preview` were shut down on 2026-06-25 and now 404. They > survive in the type union only as deprecated aliases so existing code keeps > compiling — a call to them typechecks and then fails at runtime. Use > `gemini-3.1-flash-image` / `gemini-3-pro-image` instead. > **Seedream quirks:** `watermark` defaults to **`true`** (pass > `modelOptions: { watermark: false }` for a clean image), `size` is a token > (`'1K'` | `'2K'` | `'4K'`) **or** explicit `'2048x2048'` pixels but never a > mix, and `numberOfImages` is an **upper bound** — Seedream has no `n`, so it > maps onto group-image mode and the model may return fewer. Reads > `ARK_API_KEY`. ```typescript import { generateImage } from '@tanstack/ai' import { openaiImage } from '@tanstack/ai-openai' import { geminiImage } from '@tanstack/ai-gemini' // OpenAI with quality/background options const openaiResult = await generateImage({ adapter: openaiImage('gpt-image-1'), prompt: 'A cat wearing a hat', size: '1024x1024', numberOfImages: 2, modelOptions: { quality: 'high', background: 'transparent', output_format: 'png', }, }) // Gemini native model with aspect-ratio sizes const geminiResult = await generateImage({ adapter: geminiImage('gemini-3.1-flash-image'), prompt: 'A futuristic cityscape at night', size: '16:9_4K', }) // Edit that image. geminiResult.id is the interaction id. // Send only the new prompt. The previous image stays on the server. const edited = await generateImage({ adapter: geminiImage('gemini-3.1-flash-image'), prompt: 'Make the sky orange', modelOptions: { previous_interaction_id: geminiResult.id }, }) // Gemini Imagen model. Text-to-image only. No interaction id to chain. const imagenResult = await generateImage({ adapter: geminiImage('imagen-4.0-generate-001'), prompt: 'A landscape photo', modelOptions: { aspectRatio: '16:9' }, }) ``` Gemini-native image models (`gemini-3.1-flash-image`, `gemini-3.1-flash-lite-image`, `gemini-nano-banana-2.1`, `gemini-3-pro-image`, `gemini-2.5-flash-image`) use the Interactions API. `result.id` is the interaction id. Omit `modelOptions.store` to keep the API default (`true`). `store: false` cannot be chained. Gemini keeps a stored interaction for 1 day on the free tier and 55 days on a paid tier. Set `thinkingConfig.thinkingLevel` to `'minimal'`, `'low'`, `'medium'`, or `'high'`. `thinkingConfig.thinkingBudget` throws. Imagen stays on `generateImages`. Result shape: `ImageGenerationResult` with `images` array where each entry has `b64Json?`, `url?`, and `revisedPrompt?`. OpenAI image URLs expire after 1 hour -- download or display immediately. #### Image-conditioned generation: multimodal `prompt` parts Both `generateImage()` and `generateVideo()` accept the `prompt` either as a plain string or as an ordered array of content parts (`TextPart` / `ImagePart` / `VideoPart` / `AudioPart` — the same shapes used elsewhere in TanStack AI). Part order is meaningful: natively multimodal providers (Gemini, OpenRouter) receive parts in order; named-field providers (OpenAI, fal, xAI) extract media parts and flatten the text. Prompt text is always sent verbatim — to reference inputs from the prompt, write the provider's own syntax (fal `@Image1`, OpenAI "image 1" prose); the SDK never injects or rewrites markers. Each media part may carry an optional `metadata.role` hint that adapters use to route the part to the provider-specific field. The accepted part types are narrowed per model at compile time via the adapter's input-modality map. ```typescript import { generateImage } from '@tanstack/ai' import { openaiImage } from '@tanstack/ai-openai' // Image-to-image (OpenAI gpt-image-2 / gpt-image-1, dall-e-2) await generateImage({ adapter: openaiImage('gpt-image-2'), prompt: [ { type: 'text', content: 'Turn this into a cinematic product photo' }, { type: 'image', source: { type: 'url', value: 'https://…/product.png' } }, ], }) // Multi-reference (up to 16 for gpt-image models; up to ~14 for Gemini native // — a provider limit, not enforced by the SDK) await generateImage({ adapter: openaiImage('gpt-image-2'), prompt: [ { type: 'text', content: 'Apply the second image as style to the first' }, { type: 'image', source: { type: 'url', value: 'https://…/product.png' } }, { type: 'image', source: { type: 'url', value: 'https://…/style.png' } }, ], }) // Inpaint via metadata.role === 'mask' (OpenAI gpt-image models, dall-e-2; fal mask_url) await generateImage({ adapter: openaiImage('gpt-image-2'), prompt: [ { type: 'text', content: 'Replace the masked region with a tree' }, { type: 'image', source: { type: 'url', value: 'https://…/photo.png' } }, { type: 'image', source: { type: 'url', value: 'https://…/mask.png' }, metadata: { role: 'mask' }, }, ], }) // Image-to-video (OpenAI Sora: single input_reference; fal: image_url + optional // end_image_url; OpenRouter: frame_images + input_references) import { generateVideo } from '@tanstack/ai' import { falVideo } from '@tanstack/ai-fal' await generateVideo({ adapter: falVideo('fal-ai/kling-video/v3/pro/image-to-video'), prompt: [ { type: 'image', source: { type: 'url', value: 'https://…/first.png' } }, { type: 'text', content: 'Slow cinematic push-in' }, { type: 'image', source: { type: 'url', value: 'https://…/last.png' }, metadata: { role: 'end_frame' }, }, ], }) ``` Reference images / start frames that are reused (or arrive as inline base64 on memory-constrained runtimes) can instead be uploaded once via the provider's Files adapter and referenced with `source: fileSourceFromHandle(handle)` — supported for Gemini image generation (`geminiFiles()`) and fal image/video inputs (`falFiles()`). Endpoints that require raw bytes (OpenAI `images/edits`, Sora `input_reference`, Gemini Veo) reject file sources with a clear error. See `ai-core/adapter-configuration/SKILL.md` §7. **URL inputs that require an upload throw by default.** Most adapters pass a `type: 'url'` source straight through to the provider. Three paths can't — OpenAI `images.edit()`, OpenAI Sora `input_reference`, and Gemini **Veo** — because the provider only accepts uploaded bytes (Veo also takes a `gs://` reference). For those, an HTTP(S) URL would have to be downloaded and buffered in memory, which can OOM constrained runtimes, so they **throw** on an HTTP(S) URL image input by default. Pass a `data:` URI (or `gs://` for Veo), or opt in with `allowUrlFetch: true` on the adapter config (`createOpenaiImage(model, apiKey, { allowUrlFetch: true })`, and likewise on `createOpenaiVideo` / `createGeminiVideo`). `data:` URIs never need the flag. **Role hints** (`metadata.role`): | Role | Maps to | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `'reference'` | fal `reference_image_urls`; OpenRouter video `input_references[]`; Gemini multimodal part; positional otherwise |