talking-head-video

name

talking-head-video

description

Turn a raw 口播 / talking-head selfie video plus its script into a finished vertical explainer — synced burned-in subtitles, animated knowledge graphics (charts, data cards, comparisons, flow diagrams, big-number reveals), and a circular picture-in-picture avatar that shrinks the speaker to the top-left while a graphic fills the screen during explanations. THE pipeline whenever someone has a clip of a person talking to camera and wants it "剪成成片", subtitles auto-added from a 逐字稿/脚本, the 画面 to "切成图/弹图表/做大数字特效" on key points while the "人脸/人头缩小、放到角落、变成圆形头像/画中画/小圈圈", or Remotion/ffmpeg 动态讲解图 cut into a 口播/讲解/知识科普短视频 for 视频号/抖音/ Shorts. Partial asks count: "帮我剪口播", "自动加字幕配动态图", "源视频和脚本 都有帮我合成成片". NOT for: writing a 口播脚本 (no video), a 数字人/AI 主播 reading a script, pure motion-graphics with no real person, horizontal multi-cam cutting, PPT→video, video compression, audio→text transcription, or subtitling an unrelated movie clip.

Talking-Head Explainer Video

Build a finished vertical口播 video where the speaker is full-frame by default, and during each explanation beat the speaker shrinks into a circular avatar in the top-left while an animated knowledge graphic (chart / data cards / comparison / flow / big-number) fills the screen. Subtitles are burned in and corrected against the user's verbatim script.

This skill captures a real working pipeline. Read this whole file first, then the references as needed. The design conventions exist for concrete reasons — keep them unless the user asks otherwise.

The core idea (why it's built this way)

The original talking-head video is the spine — its audio plays untouched the whole time, so sync is never an issue. You only overlay graphics + a shrunk avatar during "knowledge windows"; everywhere else it's just the full face. The graphics render as ProRes 4444 (alpha) with a short fade at their edges, so they crossfade onto the face instead of cutting to black.

full face ──┐         ┌── graphic full-frame + avatar top-left ──┐         ┌── full face
            │  cross  │   (one "knowledge window")               │  cross  │
  subtitles burned over EVERYTHING, the whole time

Prerequisites (check / install once)

• ffmpeg with libass + prores_ks (Homebrew ffmpeg has these)
• faster-whisper: python3 -m pip install faster-whisper
• Node + Remotion: the template in assets/remotion/ — npm install inside it. Remotion renders via a headless Chrome it manages.
• A CJK font libass can find (macOS: PingFang SC works out of the box).

Workflow

Set up a working dir (e.g. ~/Downloads/<project>_build/). Copy assets/remotion/ into it as remotion/. Then:

1. Inspect the source

ffprobe the clip for resolution/duration/fps. Pipeline targets vertical 1080×1920 @ 30fps. Extract a frame (ffmpeg -ss 2 -i src.mp4 -frames:v 1 f.jpg) and look at where the face sits — you need this for the avatar crop. If the source is horizontal, confirm with the user before converting (it changes everything).

2. Transcribe → timing

ffmpeg -i source.mp4 -ar 16000 -ac 1 -c:a pcm_s16le audio16k.wav
python3 scripts/transcribe.py audio16k.wav asr.json

asr.json gives word/segment timings. Trust the timing, not the text.

3. Build the segment map (the one real judgement step)

Read the user's script + asr.json and decide which time windows are knowledge windows (get a graphic + avatar) vs full-face (rhetorical beats, the hook, the conclusion — let the face carry emotion). A good rhythm alternates: don't make every second a graphic. Each knowledge window maps to one Remotion composition. Capture this as segments.json (see scripts/segments.example.json).

Pick a graphic archetype per window (see references/design-system.md → "Graphic archetypes"). Map the window's narration to the archetype's slots.

4. Build / adapt the graphics (Remotion)

In remotion/src/, each composition is one knowledge graphic. Copy the closest archetype component (RecruitChart / MedicalCards / WetDry / PromptToDrug / Edison / ThesisKeys) and rewrite its content. All of them import the shared theme.tsx — use its tokens, Backdrop, Kicker, Title, useEnter, fadeUp, useClipFade, and respect AVATAR_SAFE (keep the top-left clear). Register each in Root.tsx with durationInFrames = round(windowSeconds * 30).

Iterate fast with stills before full renders: npx remotion still src/index.ts <Comp> out.png --frame=<n>

Critical layout rule: keep all graphic content above y≈1540. The bottom ~360px is reserved for subtitles. Put payoff callouts at bottom: 380–430, not bottom: 150. (This was the #1 bug on the first build.)

5. Render graphics to alpha clips

bash scripts/render_graphics.sh remotion clips <Comp1> <Comp2> ...

Renders ProRes 4444 .mov (alpha) into clips/. Alpha is required for the edge crossfade.

6. Make the circular avatar

bash scripts/make_avatar.sh source.mp4 avatar.mov 1040:1040:0:250 0x22D3A8

Tune the crop so the whole head incl. chin sits in the circle — wider crop = smaller face. Validate on one frame first (the script prints where it wrote). Common mistake: too tight a crop clips the chin.

7. Subtitles (corrected)

python3 scripts/generate_subs.py asr.json subs.ass

Edit CORRECTIONS in the script per video — whisper mis-hears brand names / English terms / 同音字 (e.g. 施实验→湿实验, 礼来, Indeed, Verve, 胰腺癌, PPT). Skim asr.json and add the wrong→right pairs before running.

8. Composite → final

Write segments.json (source, avatar, subtitles, output, and the windows), then:

python3 scripts/build_composite.py segments.json

It generates the ffmpeg filtergraph and renders final.mp4 (graphics overlaid in their windows + avatar top-left during windows + subtitles burned + original audio). Use --dry-run to inspect the filter first.

9. QA (always)

Extract a frame from the final mp4 at the middle of every segment type and look at each:

ffmpeg -ss <t> -i final.mp4 -frames:v 1 qa_<t>.jpg

Check, per frame: avatar shows the whole face and sits in the clear top-left zone; subtitle doesn't collide with graphic content; graphic content isn't cut off; no black flash at window boundaries (mid-crossfade dark is fine). Fix, re-render only the affected clip(s), re-composite.

Editorial conventions worth keeping

• Don't make it all graphics. The face carries the hook and the emotional conclusion. Graphics are for facts/data/structure.
• One subtitle correction pass is mandatory — uncorrected ASR (wrong brand names, 同音字) instantly reads as sloppy.
• Watch the narrative framing in graphics. If the speaker's claim is a slight oversimplification you can't change (it's recorded audio), make the graphic text accurate so it doesn't contradict knowledgeable viewers. Likewise, if a segment risks an unintended read (e.g. "AI replaces humans"), add a bridging line in the graphic that steers toward the speaker's actual point.

Reusable assets in this skill

• assets/remotion/ — full Remotion project: theme.tsx (design system) + 6 archetype components + Root.tsx. Copy into the project and adapt.
• scripts/transcribe.py — faster-whisper word timings
• scripts/generate_subs.py — ASR → corrected styled .ass
• scripts/make_avatar.sh — circular alpha PiP avatar (parameterized crop/ring)
• scripts/render_graphics.sh — batch ProRes-4444 render
• scripts/build_composite.py — segments.json → ffmpeg master composite
• scripts/segments.example.json — the segment-map schema
• references/design-system.md — palette, tokens, avatar/subtitle geometry, and the graphic archetype catalog. Read before designing graphics.