storytovideo-api

name	storytovideo-api
description	Use this skill when a user wants to use storytovideo or the local storytovideo REST API to make a movie, generate a video, turn a story or script into a rendered film, produce a short film from text, render an MP4 from a prompt, or steer an in-progress video/story/script/film generation run.

1. What this API does

The storytovideo REST API is a local server that turns a text story into a rendered MP4 through a queued, asynchronous staged pipeline: analyze_story extracts structure, plan_shots creates cinematic shot plans, generate_asset and staging steps create reference imagery, generate_frame renders keyframes, generate_video renders clips, assemble builds the final MP4, and optional add_music mixes background music. Creating a run returns immediately while workers continue processing in the background.

2. Setup

The default base URL is http://localhost:3000. src/queue/queue-server.ts reads QUEUE_SERVER_PORT and defaults to 3000; src/queue/standalone-entry.ts calls startServer(), and src/queue/cluster-entry.ts starts the same server in a worker.

npm run dev

Starts the clustered queue server from src/queue/cluster-entry.ts, including backend API and web UI serving.

npm run dev

GET /api/config

Returns UI-readable feature availability: currently elevenLabsAvailable and falAvailable.

curl -sX GET http://localhost:3000/api/config \
  -H 'Accept: application/json'

GET /api/capabilities

Returns feature/concurrency capability details, including ElevenLabs availability and queue concurrency for llm, image, video, and audio.

curl -sX GET http://localhost:3000/api/capabilities \
  -H 'Accept: application/json'

GET /api/settings

Returns runtime settings persisted under the run DB directory; current shape is { "llmProvider": "anthropic" | "openai" }.

curl -sX GET http://localhost:3000/api/settings \
  -H 'Accept: application/json'

POST /api/settings

Partially updates runtime settings; unsupported values are ignored by the settings merger.

curl -sX POST http://localhost:3000/api/settings \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"llmProvider":"anthropic"}'

3. Happy path — prompt to finished movie

This is the shortest reliable flow: create a run, watch the SSE stream, poll if needed, then fetch final.mp4 or final-music.mp4 through /media/.

POST /api/runs

Creates a new asynchronous run and returns a run record with id, storyText, outputDir, status, timestamps, and options.

curl -sX POST http://localhost:3000/api/runs \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"storyText":"A tiny robot crosses a rainy city to return a lost star.","options":{"aspectRatio":"16:9","needsConversion":false,"dryRun":false,"imageBackend":"grok","assetImageBackend":"reve","videoBackend":"veo","llmProvider":"anthropic","enableVoiceReplacement":false,"fullControl":false}}'

Known options fields from src/queue/queue-server.ts near line 813:

• needsConversion: boolean; seeds story_to_script before analyze_story.
• aspectRatio: string, for example 16:9.
• dryRun: boolean.
• imageBackend: one of grok, reve, nano-banana.
• assetImageBackend: one of grok, reve, nano-banana.
• videoBackend: one of veo, veo-reve, grok, grok-quality, ltx-full, ltx-distilled, alibaba/happy-horse/reference-to-video, bytedance/seedance-2.0/reference-to-video, or bytedance/seedance-2.0/fast/reference-to-video.
• llmProvider: anthropic or openai.
• enableVoiceReplacement: honored only when ElevenLabs is available.
• fullControl: boolean; starts the run behind the manual stepping gate.

GET /api/runs/:id/events

Opens the Server-Sent Events stream for a run and replays stored events; pass Last-Event-ID to resume from a known event id.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/events \
  -H 'Accept: text/event-stream' \
  -H 'Cache-Control: no-cache'

Key SSE event names in this server:

• run_status: emitted for RunManager run:created and run:updated; payload includes status and sometimes name.
• item_started: a worker claimed an item; payload includes itemId, type, queue, and itemKey.
• item_progress: long-running item progress updates.
• item_completed: an item settled successfully; assemble completion has type: "assemble", and music completion has type: "add_music".
• item_failed: an item failed; retry details are in the queue state.
• item_redo: a redo or upstream edit superseded work and enqueued replacement items.
• item_retried: manual retry requeued a failed item.
• item_cancelled: manual cancellation.
• full_control:state: Full Control gate state changed.
• full_control:paused: a Full Control step finished and the gate paused.
• pipeline_paused: the pipeline stopped after max retries.
• cost_updated: cost summary changed.

GET /api/runs/:id

Fetches the run record plus the persisted queue state.

curl -sX GET http://localhost:3000/api/runs/RUN_ID \
  -H 'Accept: application/json'

GET /api/runs/:id/queues

Returns queue snapshots by lane and status, skipped shot info, known shots, and Full Control state.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/queues \
  -H 'Accept: application/json'

GET /api/runs/:id/costs

Returns cost summary and individual cost entries for the run.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/costs \
  -H 'Accept: application/json'

GET /api/runs/:id/graph

Returns the dependency graph used by the queue UI.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/graph \
  -H 'Accept: application/json'

GET /api/runs/:id/media/<path>

Streams a generated output file from the run directory; range requests are supported for seeking.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/media/final.mp4 \
  -H 'Accept: video/mp4' \
  -H 'Range: bytes=0-1048575' \
  -o final.mp4

Current assembled filenames are usually final.mp4 and, when music is mixed, final-music.mp4. If a caller says final-video.mp4, check the run outputs; this repo's assemble default is final.mp4.

GET /api/runs/:id/thumbnail

Serves the first generated start frame as an image thumbnail.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/thumbnail \
  -H 'Accept: image/*' \
  -o thumbnail.jpg

4. Steering mid-flight

Use these endpoints to stop, resume, manually gate stages, redo work, skip shots, and manage individual queue items.

POST /api/runs/:id/stop

Stops processors for the run; if work is in progress the run moves through stopping before stopped.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/stop \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

POST /api/runs/:id/resume

Restarts processors for a stopped, stopping, done, or completed run.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/resume \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

GET /api/runs/:id/full-control

Returns the Full Control gate state, including enabled, unlockedTypes, unlockedItemIds, nextType, peekTypes, pendingTypes, and ready/blocked counts.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/full-control \
  -H 'Accept: application/json'

POST /api/runs/:id/full-control/enable

Enables Full Control and pauses all work until a stage or item is released.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/full-control/enable \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

POST /api/runs/:id/full-control/disable

Disables Full Control so ready items can be claimed normally.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/full-control/disable \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

POST /api/runs/:id/full-control/step

Unlocks the next naturally ready work type; optionally pass targetType from current pendingTypes to skip forward stage-by-stage until that type settles.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/full-control/step \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"targetType":"generate_video"}'

POST /api/runs/:id/items/:itemId/run

Releases one ready pending item through Full Control while the stage gate stays paused; 409 means gate or dependency checks rejected the release.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/run \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

POST /api/runs/:id/assets/:assetKey/redo

Redoes the active asset-generation item, optionally overriding description, director's note, or image backend.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/assets/character%3ASophie%3Afront/redo \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"description":"Sophie with a blue raincoat","inputs":{"imageBackend":"reve"},"directorsNote":"Keep the same silhouette."}'

POST /api/runs/:id/items/:itemId/redo

Redoes a frame, video, voice, or other item, merging optional inputs, directorsNote, and durationOverride before cascading downstream.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/redo \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"inputs":{"videoBackend":"grok-quality"},"directorsNote":"Make the motion slower."}'

POST /api/runs/:id/scenes/:sceneNumber/redo

Redoes the scene's active plan_shots item, optionally with a director's note, and cascades affected downstream work.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/scenes/2/redo \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"directorsNote":"Make this scene quieter and more suspenseful."}'

POST /api/runs/:id/shots/:sceneNumber/:shotInScene/skip

Sets a shot's skipped flag and applies side effects, canceling or redoing affected items.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/shots/2/3/skip \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"skipped":true}'

POST /api/runs/:id/items/:itemId/edit

Updates an item's inputs in place without automatically redoing it.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/edit \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"inputs":{"directorsNote":"Use warmer light."}}'

POST /api/runs/:id/items/:itemId/cancel

Cancels an item and emits item_cancelled.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/cancel \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

POST /api/runs/:id/items/:itemId/retry

Manually retries a failed item without applying the usual retry limit.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/retry \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

PATCH /api/runs/:id/items/:itemId/priority

Changes an item's priority to normal or high.

curl -sX PATCH http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/priority \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"priority":"high"}'

5. Iterating on the story/script

Use the script endpoints for wholesale story text replacement, the story-view endpoint for a structured read model, and chat scopes for targeted edits. The current handler uses singular /chat/...; only /chats/active is plural.

GET /api/runs/:id/script

Returns convertedScript, original storyText, and a compact scene list.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/script \
  -H 'Accept: application/json'

PUT /api/runs/:id/script

Overwrites the story/script text, clears stale analysis scenes, redoes analyze_story, and resumes the run.

curl -sX PUT http://localhost:3000/api/runs/RUN_ID/script \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"script":"A revised visual script with three short scenes."}'

GET /api/runs/:id/story-view?cascade=true

Returns the atomic Story Page read model: run, script, scenes, assets, draft, cascade plan, groups, and fields in flight.

curl -sX GET 'http://localhost:3000/api/runs/RUN_ID/story-view?cascade=true' \
  -H 'Accept: application/json'

GET /api/runs/:id/chat/story/main

Reads the persisted chat session and current scope context for the whole story.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/story/main \
  -H 'Accept: application/json'

POST /api/runs/:id/chat/story/main

Posts UI message history to the story chat and streams the assistant response using the AI SDK UI message stream.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main \
  -H 'Accept: text/event-stream' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"id":"m1","role":"user","parts":[{"type":"text","text":"Make the ending more hopeful."}]}]}'

GET /api/runs/:id/chat/story/main/stream

Reconnects to an active chat stream; returns 204 if nothing is running.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/story/main/stream \
  -H 'Accept: text/event-stream'

POST /api/runs/:id/chat/story/main/draft

Stages a manual draft patch; accepted body fields include fields, script, scenes, music, and character-scope voiceAction where applicable.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/draft \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"script":"A cleaner story draft.","music":{"enabled":true,"volume":0.25}}'

POST /api/runs/:id/chat/story/main/apply

Applies the pending story draft; optional { "cascade": true } requests downstream invalidation where supported.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/apply \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"cascade":true}'

Story chat discard, cancel, and reset

The same story scope also supports POST /discard to clear a draft, POST /cancel to stop a running chat, and POST /reset to wipe the chat session without touching canonical story data.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/discard -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{}'
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/cancel -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{}'
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/reset -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{}'

Other chat scopes

The same GET/POST and /stream, /cancel, /apply, /discard, /draft, and /reset sub-actions exist for these patterns: /chat/scene/:sceneNumber, /chat/shot/:sceneNumber/:shotInScene, /chat/character/:characterName, /chat/location/:locationName, /chat/object/:objectName, and /chat/staging/:stagingId.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/shot/2/3 \
  -H 'Accept: application/json'

For character, location, and object, /upload also stages a reference image on the draft. The queue-server route includes staging/upload, but handleChatUpload currently accepts only character, location, and object scopes.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/character/Sophie/upload \
  -H 'Accept: application/json' \
  -H 'Content-Type: image/png' \
  --data-binary '@sophie-reference.png'

Full-cycle chat example

Post a message, reconnect if needed, then apply the draft the assistant proposed.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/scene/2 \
  -H 'Accept: text/event-stream' \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"id":"m1","role":"user","parts":[{"type":"text","text":"Make this scene take place at sunset."}]}]}'

curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/scene/2/stream \
  -H 'Accept: text/event-stream'

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/scene/2/apply \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"cascade":true}'

GET /api/runs/:id/chats/active

Lists currently active chat runners for a run.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/chats/active \
  -H 'Accept: application/json'

GET /api/runs/:id/chat-drafts

Lists chat scopes with unapplied persisted drafts.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat-drafts \
  -H 'Accept: application/json'

POST /api/runs/:id/chat-drafts/apply-all

Applies every pending chat draft, optionally filtered by scopes and with optional cascade.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat-drafts/apply-all \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"scopes":["scene","shot"],"cascade":true}'

POST /api/runs/:id/chat-drafts/discard-all

Discards all pending chat drafts, optionally filtered by scopes.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat-drafts/discard-all \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"scopes":["character","location","object"]}'

POST /api/runs/:id/chat-drafts/impact

Returns per-draft field diffs and downstream impact; defaults to timeline scopes when no scopes filter is provided.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat-drafts/impact \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"cascade":true}'

6. Reference image uploads

Uploads are local-only and unauthenticated like the rest of this server. Image validation accepts PNG, JPEG, and WebP, rejects bodies over 20 MB, rejects files under 4 KB, rejects corrupt images, and requires at least 64 by 64 pixels.

POST /api/runs/:id/upload

Uploads one raw image body for the run, pads it to the run aspect ratio, stores it under uploads/, and returns { "path": "uploads/..." }.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/upload \
  -H 'Accept: application/json' \
  -H 'Content-Type: image/png' \
  --data-binary '@reference.png'

This handler reads the raw request body; there is no multipart field name in src/queue/queue-server.ts near line 1541.

POST /api/runs/:id/assets/replace?assetKey=...

Replaces an asset image with one raw image body; current code uses the assetKey query parameter, not /assets/:assetKey/replace.

curl -sX POST 'http://localhost:3000/api/runs/RUN_ID/assets/replace?assetKey=character%3ASophie%3Afront' \
  -H 'Accept: application/json' \
  -H 'Content-Type: image/webp' \
  --data-binary '@sophie.webp'

POST /api/runs/:id/assets/replace?assetKey=... with JSON collage

Replaces an asset using one or more base64 image data URLs; multiple images are collaged into a grid before resizing.

curl -sX POST 'http://localhost:3000/api/runs/RUN_ID/assets/replace?assetKey=location%3ARooftop' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"images":["data:image/png;base64,BASE64_IMAGE_1","data:image/png;base64,BASE64_IMAGE_2"]}'

POST /api/runs/:id/chat/character/:characterName/upload

Uploads a raw image to the chat preview sandbox and stages it as pendingReferenceImage on a character draft.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/character/Sophie/upload \
  -H 'Accept: application/json' \
  -H 'Content-Type: image/png' \
  --data-binary '@sophie.png'

POST /api/runs/:id/chat/location/:locationName/upload

Uploads a raw image to the chat preview sandbox and stages it as pendingReferenceImage on a location draft.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/location/Rooftop/upload \
  -H 'Accept: application/json' \
  -H 'Content-Type: image/jpeg' \
  --data-binary '@rooftop.jpg'

POST /api/runs/:id/chat/object/:objectName/upload

Uploads a raw image to the chat preview sandbox and stages it as pendingReferenceImage on an object draft.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/object/Lost%20Star/upload \
  -H 'Accept: application/json' \
  -H 'Content-Type: image/webp' \
  --data-binary '@star.webp'

7. Final delivery

Once the run reaches done, fetch media through /media/, export the run archive, or use timeline-specific exports.

GET /api/runs/:id/media/final.mp4

Streams the assembled MP4 with Accept-Ranges: bytes; use byte ranges for seeking or partial downloads.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/media/final.mp4 \
  -H 'Accept: video/mp4' \
  -H 'Range: bytes=0-' \
  -o final.mp4

GET /api/runs/:id/export

Downloads a zip containing run-record.json and the run output directory under data/.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/export \
  -H 'Accept: application/zip' \
  -o run.zip

GET /api/runs/:id/fcpxml-export

Generates and downloads Final Cut Pro 7 XML from saved timeline_state.json, using real filesystem paths.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/fcpxml-export \
  -H 'Accept: application/xml' \
  -o timeline.xml

POST /api/runs/:id/regenerate-music

Regenerates background music from the saved timeline; if timeline-export.mp4 is missing, the handler exports it first.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/regenerate-music \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

GET /api/runs/:id/timeline

Loads saved timeline state from timeline_state.json.

curl -sX GET http://localhost:3000/api/runs/RUN_ID/timeline \
  -H 'Accept: application/json'

PUT /api/runs/:id/timeline

Saves timeline state exactly as the JSON request body.

curl -sX PUT http://localhost:3000/api/runs/RUN_ID/timeline \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"clips":[],"tracks":[],"settings":{"width":1920,"height":1080,"fps":24}}'

POST /api/runs/:id/timeline-export

Renders timeline-export.mp4 from saved timeline video clips and cross transitions.

curl -sX POST http://localhost:3000/api/runs/RUN_ID/timeline-export \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{}'

8. Gotchas

• The API is local-only and has no authentication. Do not expose it to untrusted networks.
• Run lifecycle states in src/queue/run-manager.ts are running, stopping, stopped, done, completed, and failed.
• New runs start as running. stop may set stopping while in-flight items abort, then stopped. resume moves stopped/stopping/done/completed runs back to running.
• When all active non-cancelled, non-superseded work items are completed, RunManager marks the run done and drains processors.
• completed is an accepted legacy/status value, but normal all-work-finished auto-stop currently writes done.
• In Full Control, completing the currently unlocked stage auto-pauses the gate and emits full_control:paused; you must call step or items/:itemId/run again to continue.
• pendingTypes can list upcoming stages even when some stages will never enqueue actual items for this run. For example, music-disabled runs may never create add_music work.
• Supersession is normal: upstream edits, redo, script overwrite, asset replacement, or chat apply can mark downstream items superseded and enqueue replacements.
• Do not rely on item IDs staying stable across redo. Prefer itemKey for conceptual identity, then fetch queues/graph to find the current active item.
• assets/replace can requeue affected frame items; staging replacements may requeue frames that reference that staging.
• Upload validation rejects tiny or corrupt files before they reach the pipeline: fewer than 4096 bytes, unsupported magic bytes, invalid decoder metadata, or dimensions below 64 by 64 px.
• The media endpoint confines paths to the run output directory and returns { "error": "Media file not found" } on missing files.
• Handlers generally return { "error": "..." } for 4xx and 5xx failures. 404 is most often Run not found or missing output files.
• 409 usually means a state conflict: cannot resume, cannot redo in-progress work, dependencies are not met, or Full Control rejected an item release.
• Auto-commit and git auto-pull can restart the worker. Cluster mode drains old workers gracefully and defers restarts while active items are running, but very long jobs may see a one-time pause.
• Feature flags are runtime-dependent: ElevenLabs endpoints fail if ELEVENLABS_API_KEY is not configured, and FAL-related options depend on FAL_KEY.
• The server requires the local audio service at startup. If setup is missing or health checks fail, the queue server exits rather than running degraded.