4k-vfx

# 4k-vfx Maps a video-to-video VFX move onto the Pika MCP's **Seedance 4K reference-to-video**. The user hands you a clip + the change they want; you **read EVERY frame** (locally extracted with ffmpeg and tiled into contact sheets) **and understand the audio** (speech + music + SFX + ambience), author a Seedance-faithful prompt that locks the original (face, gestures, camera move) and time-codes the change to the right beat, then re-render the **same shot** in 4K with the VFX baked in. **One run produces one 4K clip** for one requested change. ## How it works — why reading EVERY frame (and the audio) is the mechanism (do not skip the reading pass) Seedance treats the reference video as a **motion/style anchor, NOT a pixel-locked base** — it does not copy the input frame-for-frame, it re-generates the shot guided by your prompt. So fidelity comes from the **PROMPT**: the more exhaustively the prompt describes the original (subject identity, exact gestures, camera move, framing, lighting, palette, wardrobe) **and the audio** (dialogue + lip-sync, music/SFX beats), the more the 4K output reads as the *same* shot with the change layered on. **Reading EVERY frame — not a sample — is how you build that exhaustive description:** you locally extract all frames with ffmpeg, tile them 25-to-a-contact-sheet (5×5), and analyze every sheet so the prompt is built from the *entire* timeline rather than a handful of stills. Skip the read (or read only a sample) and the output drifts — a different face, a different camera move, a different room, a missed motion beat. The reading pass — all frames + the audio — IS the skill. ## Prerequisites pika MCP available, and **local `ffmpeg`** in the run environment (used to extract every frame and build the contact sheets). Two inputs from the user: **(1) a video clip** (a local file or a public URL) and **(2) the VFX change** they want made to it. Tools used: `upload_asset`, `probe_media`, `analyze_media`, `transcribe_audio`, `estimate_cost`, `generate_reference_video`, `task_status` (frame extraction + tiling is done locally with ffmpeg, not an MCP tool). ## Stage 0 — Gather inputs (settle this first) You need exactly two things before generating: 1. **The video** — the original clip to transform. If they gave a local file, you will upload it (Step 1); if they gave a public URL, keep it. 2. **The change / VFX** — what should be different in the output (e.g. "turn the plaza into an open desert on a finger snap", "make it snow", "set the room on fire behind me", "morph my jacket into glowing armor"). If **either** is missing, ask for it before doing anything else — do not invent a change, and do not proceed on a clip you cannot reach. Once both are in hand, say in one line what you're about to do ("Reading every frame of your clip to build the desert-on-snap 4K prompt…") and run Steps 1–5 without further check-ins — but **stop at the Step 6 agreement gate before generating**: the 4K render is the expensive, irreversible step, and it only fires after the user approves the prompt and cost. ## Step 1 — Upload the clip (`upload_asset`) Real video must be uploaded — inline base64 only works for tiny assets (<~3MB), so any actual clip goes through the presigned-upload path. 1. Call `upload_asset` with `filename`, `mime_type` (`video/mp4` for mp4), and `size_bytes` (the file's byte size). 2. PUT the raw bytes to the returned `presigned_url`. 3. Keep the returned `public_url` → `state.clip_url`. If the user already gave a **public URL**, skip this step and set `state.clip_url` to it. ## Step 2 — Probe (`probe_media`) Call `probe_media` on `state.clip_url` to read **duration, fps, dimensions, and aspect ratio**. Use these to: (a) sanity-check the all-frame extraction in Step 3 (duration × fps ≈ total frame count ≈ 25 × number of sheets), and (b) choose the output `aspect_ratio` and `duration` here, matched to the source — the Step-6 gate quotes this choice to the user and Step 7 fires with it. ## Step 3 — Extract EVERY frame locally and tile into contact sheets (ffmpeg) Read the **whole** video, not a sample. Use local ffmpeg to extract **every** frame at the source rate (no drops, no exceptions), then tile them into legible contact sheets at **25 frames per sheet (5×5 grid)** so each `analyze_media` call covers 25 consecutive frames at once. Work in a **relative** working directory (do not use an absolute system temp path): ``` mkdir -p frames sheets ffmpeg -i INPUT -vsync 0 frames/f_%05d.png # every frame, no drops ``` Then build the 25-per-sheet contact sheets (5×5), scaling each frame down so the sheet is legible and stamping the frame index onto each tile: ``` ffmpeg -i INPUT -vf "scale=480:-1,drawtext=text='%{n}':x=4:y=4:fontsize=20:fontcolor=yellow:box=1:boxcolor=black@0.5,tile=5x5" sheets/sheet_%03d.png ``` `drawtext=text='%{n}'` stamps the frame index on each tile so the analysis can reference exact frames; `tile=5x5` groups 25 consecutive frames per sheet. If `drawtext` is unavailable in the local ffmpeg build, fall back to plain `scale=480:-1,tile=5x5`. `INPUT` is the local clip (or a local copy of the URL). Confirm the sheet count lines up with the probe in Step 2 — **every frame must land on a sheet, no exceptions.** ## Step 4 — Read EVERY sheet + understand the audio (`upload_asset`, `analyze_media`, `transcribe_audio`) This is the fidelity step, and it has two mandatory halves: **read all the frames** and **understand the audio**. Neither is optional. ### 4a — Read all the frames (every sheet) For **each** contact sheet from Step 3 (cover ALL of them — every frame is on a sheet, no exceptions): upload it with `upload_asset` (the sheets are images, so upload to get a URL), then call `analyze_media` on that sheet URL with a `query` that reads the **full progression across those 25 frames** — pulling out everything the prompt must lock: - **Subject / identity** — who/what is in frame; face, hair, build, distinguishing features. - **Exact motion & gestures** — what the subject does, beat by beat, referencing the stamped frame indices (e.g. "raises right hand around frame 0048, snaps fingers around frame 0072"). - **Camera move** — static / pan / push-in / handheld / orbit, and its timing. - **Framing** — shot size (close/medium/wide), subject position, headroom. - **Lighting** — direction, hardness, color temperature, time of day. - **Palette** — dominant colors, mood. - **Wardrobe / props / setting** — clothing, objects, background. Carry the read across **all** sheets so the **entire timeline** is understood start-to-end, not just a sampled middle. ### 4b — Understand the audio (mandatory, not just speech) Always read the audio — it is required, not optional. Two passes: - `transcribe_audio` for **speech + timestamps** (the words and when they're said), so the prompt can preserve dialogue and lip-sync beats. - `analyze_media` on the audio (or the video URL) to understand the **NON-speech audio too** — music, sound effects, ambience, tone, and rhythm/beats. Fold the audio understanding into the prompt where it matters — preserve dialogue + lip-sync if anyone speaks, and **time the VFX change to an audio beat / sound cue** when the change should land on the music or a sound (e.g. snap, hit, downbeat) rather than a bare timestamp. Assemble the all-frame read **and** the audio read into a single faithful description of the original — that description is the raw material for Step 5. ## Step 5 — Write the Seedance prompt (the authoring step, NEVER skip it) Compose ONE prompt = **a faithful, exhaustive description of the original** (the locked elements) **+ the user's requested change, time-coded** + a reference to the input clip via the **`@Video1`** token. Be thorough: every detail you read in Step 4 that you omit is a detail Seedance is free to change. **Template — fill the `[SLOTS]` from your Step-4 read:** ``` Re-render this exact shot @Video1 in cinematic 4K. LOCK the original: [SUBJECT/IDENTITY — face, hair, build, wardrobe], performing [EXACT GESTURES, beat by beat]. Keep the SAME camera move ([CAMERA: static / pan / push-in / handheld], [timing]), the SAME framing ([SHOT SIZE + subject position]), the SAME lighting ([direction, hardness, color temp, time of day]) and the SAME palette ([dominant colors / mood]). CHANGE: [the VFX], time-coded — at [t0]–[t1]s [what the scene looks like before the change], then at [t2]s [the change triggers / VFX appears] and [how the scene reads after]. Everything not described by the change stays identical to @Video1. Photorealistic, high detail, consistent identity throughout. ``` Time-code the change so Seedance knows *when* it happens relative to the locked motion (e.g. "at 0–3s the plaza is unchanged as the subject raises their hand; at 3s on the finger snap the plaza dissolves into an open desert, sand and heat-haze replacing the buildings while the subject, camera move, and framing stay identical"). The stronger and more specific the time-coded change clause, the less Seedance ignores it. ## Step 6 — Agreement gate (`estimate_cost`) — MANDATORY before generating The 4K render is the expensive step, and once fired it can't be un-spent. **Never call `generate_reference_video` without explicit user approval in this session.** Present, in one message: 1. **The full Step-5 prompt** — verbatim, so the user can catch a wrong detail (face, gesture, timing) before it costs money. 2. **The generation recipe** — provider `seedance`, model `standard`, resolution `4k`, plus the `aspect_ratio` and `duration` you chose from the Step-2 probe. 3. **The estimated cost** — call `estimate_cost` for the Step-7 `generate_reference_video` call and quote the result. If the estimate errors, say the cost is unknown and flag that 4K standard is the top-price tier — do not silently skip the number. Then **wait**. Three outcomes: - **Approve** → proceed to Step 7 unchanged. - **Revise** — the user corrects a detail or the change clause → update the Step-5 prompt (re-reading specific sheets if the correction demands it) and re-present the gate. - **Abort / no answer** → do not generate. Never treat silence as approval. Only skip this gate if the user **explicitly pre-authorized the spend in this session** ("just run it, don't ask", a standing instruction to render without confirmation). Having provided the clip and the change is NOT pre-authorization — that's just Stage 0 input. ## Step 7 — Generate (Seedance 4K reference-to-video) (`generate_reference_video`) Call `generate_reference_video` with the **exact** 4K recipe: - `provider`: **`seedance`** - `seedance_model`: **`standard`** — REQUIRED for 4k (fast/mini cap at 720p). - `resolution`: **`4k`** - `reference_videos`: **`[state.clip_url]`** — the original footage as the motion/style anchor (Seedance accepts up to 3 videos). - `reference_images`: optional — style/creature/identity anchors (e.g. a generated reference image), only if you have one. - `prompt`: the Step-5 prompt (uses the **`@Video1`** token to reference the input clip; no max length, so be exhaustive). - `aspect_ratio`: **`auto`** (or match the source from Step 2). - `duration`: matched to the source, OR set `auto_duration`: **`true`**. **Do NOT pass `negative_prompt`** (not supported on seedance — the call is rejected) and **do NOT pass any fps param** (it doesn't exist). ## Step 8 — Deliver If the call returns inline, you have the 4K video URL. If it returns `{ task_id, status: running/background }`, poll `task_status(task_id)` in a tight loop until `completed` / `failed` / `cancelled`. On completion, return the **4K video URL** to the user. Note: the output CDN may be egress-blocked for in-session download — so deliver the URL itself rather than trying to fetch the bytes locally. ## Failure modes | symptom | cause → fix | |---|---| | Clip too long / too large | Upload still works for large files; if Seedance rejects on duration, trim the source to the segment that matters, or set `auto_duration: true` and match a shorter `duration`. | | No face / subject found in the read | A sheet may not show the subject clearly — re-run `analyze_media` on the specific sheet(s) covering the frames where the subject is visible (use the stamped frame indices), or rebuild the sheets without downscaling so detail is preserved. Do NOT drop to a sampled extraction — every frame stays on a sheet. | | ffmpeg missing / `drawtext` unavailable | The run environment needs local `ffmpeg`; if the build lacks the `drawtext` filter, fall back to plain `scale=480:-1,tile=5x5` (sheets without stamped indices — still read every frame). | | Identity / shot drift (different face, room, or camera move in the output) | The prompt description was not exhaustive enough. Go back to Step 4–5 and add the missing locked details (more on face, gestures, camera move, lighting) from the contact-sheet read — Seedance only preserves what the prompt names. Re-render = new spend: re-pass the Step-6 gate with the revised prompt before firing again. | | 4k rejected / output downgraded | Ensure `seedance_model: standard` (fast/mini cap at 720p) and `resolution: 4k`. If the wrong-param call already completed (and was charged), the corrected call is a second render — re-pass the Step-6 gate before re-firing; if the call was rejected outright (nothing charged), re-fire the corrected call under the original approval. | | Speech / audio not preserved or change off-beat | Run BOTH `transcribe_audio` (speech + timing) and `analyze_media` on the audio (music/SFX/ambience), then describe the dialogue + lip-movement beats and pin the time-coded change to the right audio cue/beat in the prompt so the re-render stays in sync. | | Seedance ignores the requested change | Strengthen the time-coded change clause in Step 5 — make it more specific and explicitly time-anchored ("at Ns the change triggers"). The revised prompt is a new, never-approved prompt and the re-render is new spend: re-pass the Step-6 gate before firing again. | | Job fails after a long run | If it ran several minutes before failing, the content already cleared the safety pass — the failure is infrastructural; re-fire the same call (the user's Step-6 approval covers retrying the identical, already-approved call — no need to re-gate). | ## One-liner recap Extract EVERY frame locally with ffmpeg → tile into 25-per-sheet (5×5) contact sheets and analyze every sheet → understand the audio (transcribe speech + analyze music/SFX/ambience) → write an exhaustive Seedance-faithful prompt that locks the original and time-codes the change to the right audio beat → **agreement gate: show the prompt + recipe + `estimate_cost` and wait for approval** → re-render the same shot in 4K via `generate_reference_video` (`seedance` / `standard` / `4k`, original as `reference_videos`, `@Video1` in the prompt, no `negative_prompt`).