| name | storytovideo-api |
| description | Use this skill when a user wants to use storytovideo or the local storytovideo REST API to make a movie, generate a video, turn a story or script into a rendered film, produce a short film from text, render an MP4 from a prompt, or steer an in-progress video/story/script/film generation run. |
1. What this API does
The storytovideo REST API is a local server that turns a text story into a rendered MP4 through a queued, asynchronous staged pipeline: analyze_story extracts structure, plan_shots creates cinematic shot plans, generate_asset and staging steps create reference imagery, generate_frame renders keyframes, generate_video renders clips, assemble builds the final MP4, and optional add_music mixes background music. Creating a run returns immediately while workers continue processing in the background.
2. Setup
The default base URL is http://localhost:3000. src/queue/queue-server.ts reads QUEUE_SERVER_PORT and defaults to 3000; src/queue/standalone-entry.ts calls startServer(), and src/queue/cluster-entry.ts starts the same server in a worker.
npm run dev
Starts the clustered queue server from src/queue/cluster-entry.ts, including backend API and web UI serving.
npm run dev
GET /api/config
Returns UI-readable feature availability: currently elevenLabsAvailable and falAvailable.
curl -sX GET http://localhost:3000/api/config \
-H 'Accept: application/json'
GET /api/capabilities
Returns feature/concurrency capability details, including ElevenLabs availability and queue concurrency for llm, image, video, and audio.
curl -sX GET http://localhost:3000/api/capabilities \
-H 'Accept: application/json'
GET /api/settings
Returns runtime settings persisted under the run DB directory; current shape is { "llmProvider": "anthropic" | "openai" }.
curl -sX GET http://localhost:3000/api/settings \
-H 'Accept: application/json'
POST /api/settings
Partially updates runtime settings; unsupported values are ignored by the settings merger.
curl -sX POST http://localhost:3000/api/settings \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"llmProvider":"anthropic"}'
3. Happy path — prompt to finished movie
This is the shortest reliable flow: create a run, watch the SSE stream, poll if needed, then fetch final.mp4 or final-music.mp4 through /media/.
POST /api/runs
Creates a new asynchronous run and returns a run record with id, storyText, outputDir, status, timestamps, and options.
curl -sX POST http://localhost:3000/api/runs \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"storyText":"A tiny robot crosses a rainy city to return a lost star.","options":{"aspectRatio":"16:9","needsConversion":false,"dryRun":false,"imageBackend":"grok","assetImageBackend":"reve","videoBackend":"veo","llmProvider":"anthropic","enableVoiceReplacement":false,"fullControl":false}}'
Known options fields from src/queue/queue-server.ts near line 813:
needsConversion: boolean; seeds story_to_script before analyze_story.aspectRatio: string, for example 16:9.dryRun: boolean.imageBackend: one of grok, reve, nano-banana.assetImageBackend: one of grok, reve, nano-banana.videoBackend: one of veo, veo-reve, grok, grok-quality, ltx-full, ltx-distilled, alibaba/happy-horse/reference-to-video, bytedance/seedance-2.0/reference-to-video, or bytedance/seedance-2.0/fast/reference-to-video.llmProvider: anthropic or openai.enableVoiceReplacement: honored only when ElevenLabs is available.fullControl: boolean; starts the run behind the manual stepping gate.
GET /api/runs/:id/events
Opens the Server-Sent Events stream for a run and replays stored events; pass Last-Event-ID to resume from a known event id.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/events \
-H 'Accept: text/event-stream' \
-H 'Cache-Control: no-cache'
Key SSE event names in this server:
run_status: emitted for RunManager run:created and run:updated; payload includes status and sometimes name.item_started: a worker claimed an item; payload includes itemId, type, queue, and itemKey.item_progress: long-running item progress updates.item_completed: an item settled successfully; assemble completion has type: "assemble", and music completion has type: "add_music".item_failed: an item failed; retry details are in the queue state.item_redo: a redo or upstream edit superseded work and enqueued replacement items.item_retried: manual retry requeued a failed item.item_cancelled: manual cancellation.full_control:state: Full Control gate state changed.full_control:paused: a Full Control step finished and the gate paused.pipeline_paused: the pipeline stopped after max retries.cost_updated: cost summary changed.
GET /api/runs/:id
Fetches the run record plus the persisted queue state.
curl -sX GET http://localhost:3000/api/runs/RUN_ID \
-H 'Accept: application/json'
GET /api/runs/:id/queues
Returns queue snapshots by lane and status, skipped shot info, known shots, and Full Control state.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/queues \
-H 'Accept: application/json'
GET /api/runs/:id/costs
Returns cost summary and individual cost entries for the run.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/costs \
-H 'Accept: application/json'
GET /api/runs/:id/graph
Returns the dependency graph used by the queue UI.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/graph \
-H 'Accept: application/json'
GET /api/runs/:id/media/<path>
Streams a generated output file from the run directory; range requests are supported for seeking.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/media/final.mp4 \
-H 'Accept: video/mp4' \
-H 'Range: bytes=0-1048575' \
-o final.mp4
Current assembled filenames are usually final.mp4 and, when music is mixed, final-music.mp4. If a caller says final-video.mp4, check the run outputs; this repo's assemble default is final.mp4.
GET /api/runs/:id/thumbnail
Serves the first generated start frame as an image thumbnail.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/thumbnail \
-H 'Accept: image/*' \
-o thumbnail.jpg
4. Steering mid-flight
Use these endpoints to stop, resume, manually gate stages, redo work, skip shots, and manage individual queue items.
POST /api/runs/:id/stop
Stops processors for the run; if work is in progress the run moves through stopping before stopped.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/stop \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
POST /api/runs/:id/resume
Restarts processors for a stopped, stopping, done, or completed run.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/resume \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
GET /api/runs/:id/full-control
Returns the Full Control gate state, including enabled, unlockedTypes, unlockedItemIds, nextType, peekTypes, pendingTypes, and ready/blocked counts.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/full-control \
-H 'Accept: application/json'
POST /api/runs/:id/full-control/enable
Enables Full Control and pauses all work until a stage or item is released.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/full-control/enable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
POST /api/runs/:id/full-control/disable
Disables Full Control so ready items can be claimed normally.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/full-control/disable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
POST /api/runs/:id/full-control/step
Unlocks the next naturally ready work type; optionally pass targetType from current pendingTypes to skip forward stage-by-stage until that type settles.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/full-control/step \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"targetType":"generate_video"}'
POST /api/runs/:id/items/:itemId/run
Releases one ready pending item through Full Control while the stage gate stays paused; 409 means gate or dependency checks rejected the release.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/run \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
POST /api/runs/:id/assets/:assetKey/redo
Redoes the active asset-generation item, optionally overriding description, director's note, or image backend.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/assets/character%3ASophie%3Afront/redo \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"description":"Sophie with a blue raincoat","inputs":{"imageBackend":"reve"},"directorsNote":"Keep the same silhouette."}'
POST /api/runs/:id/items/:itemId/redo
Redoes a frame, video, voice, or other item, merging optional inputs, directorsNote, and durationOverride before cascading downstream.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/redo \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"inputs":{"videoBackend":"grok-quality"},"directorsNote":"Make the motion slower."}'
POST /api/runs/:id/scenes/:sceneNumber/redo
Redoes the scene's active plan_shots item, optionally with a director's note, and cascades affected downstream work.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/scenes/2/redo \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"directorsNote":"Make this scene quieter and more suspenseful."}'
POST /api/runs/:id/shots/:sceneNumber/:shotInScene/skip
Sets a shot's skipped flag and applies side effects, canceling or redoing affected items.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/shots/2/3/skip \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"skipped":true}'
POST /api/runs/:id/items/:itemId/edit
Updates an item's inputs in place without automatically redoing it.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/edit \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"inputs":{"directorsNote":"Use warmer light."}}'
POST /api/runs/:id/items/:itemId/cancel
Cancels an item and emits item_cancelled.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/cancel \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
POST /api/runs/:id/items/:itemId/retry
Manually retries a failed item without applying the usual retry limit.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/retry \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
PATCH /api/runs/:id/items/:itemId/priority
Changes an item's priority to normal or high.
curl -sX PATCH http://localhost:3000/api/runs/RUN_ID/items/ITEM_ID/priority \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"priority":"high"}'
5. Iterating on the story/script
Use the script endpoints for wholesale story text replacement, the story-view endpoint for a structured read model, and chat scopes for targeted edits. The current handler uses singular /chat/...; only /chats/active is plural.
GET /api/runs/:id/script
Returns convertedScript, original storyText, and a compact scene list.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/script \
-H 'Accept: application/json'
PUT /api/runs/:id/script
Overwrites the story/script text, clears stale analysis scenes, redoes analyze_story, and resumes the run.
curl -sX PUT http://localhost:3000/api/runs/RUN_ID/script \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"script":"A revised visual script with three short scenes."}'
GET /api/runs/:id/story-view?cascade=true
Returns the atomic Story Page read model: run, script, scenes, assets, draft, cascade plan, groups, and fields in flight.
curl -sX GET 'http://localhost:3000/api/runs/RUN_ID/story-view?cascade=true' \
-H 'Accept: application/json'
GET /api/runs/:id/chat/story/main
Reads the persisted chat session and current scope context for the whole story.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/story/main \
-H 'Accept: application/json'
POST /api/runs/:id/chat/story/main
Posts UI message history to the story chat and streams the assistant response using the AI SDK UI message stream.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main \
-H 'Accept: text/event-stream' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"id":"m1","role":"user","parts":[{"type":"text","text":"Make the ending more hopeful."}]}]}'
GET /api/runs/:id/chat/story/main/stream
Reconnects to an active chat stream; returns 204 if nothing is running.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/story/main/stream \
-H 'Accept: text/event-stream'
POST /api/runs/:id/chat/story/main/draft
Stages a manual draft patch; accepted body fields include fields, script, scenes, music, and character-scope voiceAction where applicable.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/draft \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"script":"A cleaner story draft.","music":{"enabled":true,"volume":0.25}}'
POST /api/runs/:id/chat/story/main/apply
Applies the pending story draft; optional { "cascade": true } requests downstream invalidation where supported.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/apply \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"cascade":true}'
Story chat discard, cancel, and reset
The same story scope also supports POST /discard to clear a draft, POST /cancel to stop a running chat, and POST /reset to wipe the chat session without touching canonical story data.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/discard -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{}'
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/cancel -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{}'
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/story/main/reset -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{}'
Other chat scopes
The same GET/POST and /stream, /cancel, /apply, /discard, /draft, and /reset sub-actions exist for these patterns: /chat/scene/:sceneNumber, /chat/shot/:sceneNumber/:shotInScene, /chat/character/:characterName, /chat/location/:locationName, /chat/object/:objectName, and /chat/staging/:stagingId.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/shot/2/3 \
-H 'Accept: application/json'
For character, location, and object, /upload also stages a reference image on the draft. The queue-server route includes staging/upload, but handleChatUpload currently accepts only character, location, and object scopes.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/character/Sophie/upload \
-H 'Accept: application/json' \
-H 'Content-Type: image/png' \
--data-binary '@sophie-reference.png'
Full-cycle chat example
Post a message, reconnect if needed, then apply the draft the assistant proposed.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/scene/2 \
-H 'Accept: text/event-stream' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"id":"m1","role":"user","parts":[{"type":"text","text":"Make this scene take place at sunset."}]}]}'
curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat/scene/2/stream \
-H 'Accept: text/event-stream'
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/scene/2/apply \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"cascade":true}'
GET /api/runs/:id/chats/active
Lists currently active chat runners for a run.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/chats/active \
-H 'Accept: application/json'
GET /api/runs/:id/chat-drafts
Lists chat scopes with unapplied persisted drafts.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/chat-drafts \
-H 'Accept: application/json'
POST /api/runs/:id/chat-drafts/apply-all
Applies every pending chat draft, optionally filtered by scopes and with optional cascade.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat-drafts/apply-all \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"scopes":["scene","shot"],"cascade":true}'
POST /api/runs/:id/chat-drafts/discard-all
Discards all pending chat drafts, optionally filtered by scopes.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat-drafts/discard-all \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"scopes":["character","location","object"]}'
POST /api/runs/:id/chat-drafts/impact
Returns per-draft field diffs and downstream impact; defaults to timeline scopes when no scopes filter is provided.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat-drafts/impact \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"cascade":true}'
6. Reference image uploads
Uploads are local-only and unauthenticated like the rest of this server. Image validation accepts PNG, JPEG, and WebP, rejects bodies over 20 MB, rejects files under 4 KB, rejects corrupt images, and requires at least 64 by 64 pixels.
POST /api/runs/:id/upload
Uploads one raw image body for the run, pads it to the run aspect ratio, stores it under uploads/, and returns { "path": "uploads/..." }.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/upload \
-H 'Accept: application/json' \
-H 'Content-Type: image/png' \
--data-binary '@reference.png'
This handler reads the raw request body; there is no multipart field name in src/queue/queue-server.ts near line 1541.
POST /api/runs/:id/assets/replace?assetKey=...
Replaces an asset image with one raw image body; current code uses the assetKey query parameter, not /assets/:assetKey/replace.
curl -sX POST 'http://localhost:3000/api/runs/RUN_ID/assets/replace?assetKey=character%3ASophie%3Afront' \
-H 'Accept: application/json' \
-H 'Content-Type: image/webp' \
--data-binary '@sophie.webp'
POST /api/runs/:id/assets/replace?assetKey=... with JSON collage
Replaces an asset using one or more base64 image data URLs; multiple images are collaged into a grid before resizing.
curl -sX POST 'http://localhost:3000/api/runs/RUN_ID/assets/replace?assetKey=location%3ARooftop' \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"images":["data:image/png;base64,BASE64_IMAGE_1","data:image/png;base64,BASE64_IMAGE_2"]}'
POST /api/runs/:id/chat/character/:characterName/upload
Uploads a raw image to the chat preview sandbox and stages it as pendingReferenceImage on a character draft.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/character/Sophie/upload \
-H 'Accept: application/json' \
-H 'Content-Type: image/png' \
--data-binary '@sophie.png'
POST /api/runs/:id/chat/location/:locationName/upload
Uploads a raw image to the chat preview sandbox and stages it as pendingReferenceImage on a location draft.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/location/Rooftop/upload \
-H 'Accept: application/json' \
-H 'Content-Type: image/jpeg' \
--data-binary '@rooftop.jpg'
POST /api/runs/:id/chat/object/:objectName/upload
Uploads a raw image to the chat preview sandbox and stages it as pendingReferenceImage on an object draft.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/chat/object/Lost%20Star/upload \
-H 'Accept: application/json' \
-H 'Content-Type: image/webp' \
--data-binary '@star.webp'
7. Final delivery
Once the run reaches done, fetch media through /media/, export the run archive, or use timeline-specific exports.
GET /api/runs/:id/media/final.mp4
Streams the assembled MP4 with Accept-Ranges: bytes; use byte ranges for seeking or partial downloads.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/media/final.mp4 \
-H 'Accept: video/mp4' \
-H 'Range: bytes=0-' \
-o final.mp4
GET /api/runs/:id/export
Downloads a zip containing run-record.json and the run output directory under data/.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/export \
-H 'Accept: application/zip' \
-o run.zip
GET /api/runs/:id/fcpxml-export
Generates and downloads Final Cut Pro 7 XML from saved timeline_state.json, using real filesystem paths.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/fcpxml-export \
-H 'Accept: application/xml' \
-o timeline.xml
POST /api/runs/:id/regenerate-music
Regenerates background music from the saved timeline; if timeline-export.mp4 is missing, the handler exports it first.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/regenerate-music \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
GET /api/runs/:id/timeline
Loads saved timeline state from timeline_state.json.
curl -sX GET http://localhost:3000/api/runs/RUN_ID/timeline \
-H 'Accept: application/json'
PUT /api/runs/:id/timeline
Saves timeline state exactly as the JSON request body.
curl -sX PUT http://localhost:3000/api/runs/RUN_ID/timeline \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"clips":[],"tracks":[],"settings":{"width":1920,"height":1080,"fps":24}}'
POST /api/runs/:id/timeline-export
Renders timeline-export.mp4 from saved timeline video clips and cross transitions.
curl -sX POST http://localhost:3000/api/runs/RUN_ID/timeline-export \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{}'
8. Gotchas
- The API is local-only and has no authentication. Do not expose it to untrusted networks.
- Run lifecycle states in
src/queue/run-manager.ts are running, stopping, stopped, done, completed, and failed.
- New runs start as
running. stop may set stopping while in-flight items abort, then stopped. resume moves stopped/stopping/done/completed runs back to running.
- When all active non-cancelled, non-superseded work items are
completed, RunManager marks the run done and drains processors.
completed is an accepted legacy/status value, but normal all-work-finished auto-stop currently writes done.- In Full Control, completing the currently unlocked stage auto-pauses the gate and emits
full_control:paused; you must call step or items/:itemId/run again to continue.
pendingTypes can list upcoming stages even when some stages will never enqueue actual items for this run. For example, music-disabled runs may never create add_music work.- Supersession is normal: upstream edits, redo, script overwrite, asset replacement, or chat apply can mark downstream items
superseded and enqueue replacements.
- Do not rely on item IDs staying stable across redo. Prefer
itemKey for conceptual identity, then fetch queues/graph to find the current active item.
assets/replace can requeue affected frame items; staging replacements may requeue frames that reference that staging.- Upload validation rejects tiny or corrupt files before they reach the pipeline: fewer than 4096 bytes, unsupported magic bytes, invalid decoder metadata, or dimensions below 64 by 64 px.
- The media endpoint confines paths to the run output directory and returns
{ "error": "Media file not found" } on missing files.
- Handlers generally return
{ "error": "..." } for 4xx and 5xx failures. 404 is most often Run not found or missing output files.
409 usually means a state conflict: cannot resume, cannot redo in-progress work, dependencies are not met, or Full Control rejected an item release.- Auto-commit and git auto-pull can restart the worker. Cluster mode drains old workers gracefully and defers restarts while active items are running, but very long jobs may see a one-time pause.
- Feature flags are runtime-dependent: ElevenLabs endpoints fail if
ELEVENLABS_API_KEY is not configured, and FAL-related options depend on FAL_KEY.
- The server requires the local audio service at startup. If setup is missing or health checks fail, the queue server exits rather than running degraded.
