raas-video-pipeline

name

raas-video-pipeline

description

End-to-end pipeline for RaaS short-form video production. Use this skill for every new video project — from script through VO generation, Whisper canon extraction, background image generation, OCR highlight plans, HeyGen avatar clips, Remotion scene assembly, and final render. Covers audio-first law, hero/trail split, voice config, multi-template composition (TextCard + SplitScreenAvatar + ImageKaraoke), BGM layer, BackgroundZone, HeadlineHighlighter, and QA gates. Triggers: new video, generate short, create raas short, build video, new project, raas content, make a video, short-00X, image-led video, highlight words, OCR karaoke.

RaaS Video Pipeline

End-to-end production pipeline for all RaaS short-form videos (9:16, YouTube Shorts / Instagram Reels).

Core law: Audio is SSOT. Words dictate scenes. Never the other way around.

---

Architecture

RaaSComposition.tsx              ← single composition for ALL videos
  ├── Layer 1: Voiceover         ← script.voiceover (primary, vol=1)
  ├── Layer 2: BGM               ← script.bgm (ducked, configurable)
  └── Layer 3: Scenes            ← Sequence per scene
        ├── template='TextCard'              → SceneRenderer (TextCard engine)
        ├── template='SplitScreenAvatar'     → SplitScreenAvatarTemplate
        └── template='ImageKaraoke'          → ImageKaraokeTemplate

One composition. Props = JSON data per video. Templates = rendering components.

---

Templates

Template	File	Use case
TextCard	RaaSComposition.tsx SceneRenderer	Hero (Bolt Classic) + Trail (TikTok karaoke)
SplitScreenAvatar	templates/split-screen-avatar/SplitScreenAvatarTemplate.tsx	62% content top / 38% avatar bottom. Supports OCR karaoke highlights in top zone.
ImageKaraoke	templates/image-karaoke/ImageKaraokeTemplate.tsx	Full-screen image with OCR word highlights synced to VO. Fallback TikTok captions when no OCR text.

Template Selection Guide

Content type	Template	Notes
AI-generated background + commentary	TextCard	Hero/trail subtitles
Article/screenshot + avatar talking	SplitScreenAvatar	Add karaoke_plan for OCR highlights in top zone
Full-screen screenshot/headline	ImageKaraoke	Words highlight directly on image
Physical product shot (no text)	ImageKaraoke	Auto-falls back to TikTok captions
Tagline/brand card + avatar	SplitScreenAvatar	karaoke_plan highlights the tagline words

Default pattern

Alternate: SplitScreenAvatar → ImageKaraoke → SplitScreenAvatar → ImageKaraoke... (Or use TextCard when no source image is available.)

---

Key Files

File	Purpose
remotion/src/RaaSComposition.tsx	Single composition — audio stack + scene router
remotion/src/templates/split-screen-avatar/SplitScreenAvatarTemplate.tsx	SplitScreen + avatar + optional OCR highlights
remotion/src/templates/image-karaoke/ImageKaraokeTemplate.tsx	Full-image karaoke template
remotion/src/components/BackgroundZone.tsx	Shared background primitive (fit/dim/position)
remotion/src/components/HeadlineHighlighter.tsx	Phrase-level highlight overlay (used by TextCard scenes)
remotion/src/compositions/registry.tsx	All composition registrations
remotion/src/types.ts	ShortScript + Scene types
voices/bolt-siw1-config.json	Locked voice settings + pronunciation corrections
tools/generate_vo.py	VO generation via ElevenLabs
tools/generate_avatar.py	Avatar clip generation via HeyGen API
tools/generate_images.py	Scene background generation via Replicate (flux-dev)
tools/extract_ocr_layout.py	Tesseract OCR word-level box extraction
tools/generate_highlight_plan.py	Phrase-to-OCR-box highlight plan generator
tools/generate_word_layover_plan.py	Deterministic per-word OCR layover plan generator
remotion/public/assets/bgm/bgm-cinematic-tech-ambient.mp3	Default BGM track

---

Script Schema

Standard scenes (TextCard / SplitScreenAvatar)

{
  "scene_id": "scene_001",
  "hero": "Robotics as a Service.",
  "trail": "This is how you enter:",
  "hero_split_at": 4,
  "hero_position": "end",
  "template": "SplitScreenAvatar"
}

Image-grounded scenes (ImageKaraoke / SplitScreenAvatar with highlights)

{
  "scene_id": "scene_002",
  "source_image": "assets/images/src/img-02.jpg",
  "template": "ImageKaraoke",
  "vo_line": "India's Robotics Breakthrough. A Startup That Handles Objects Like Humans.",
  "highlight_words": ["India's", "Robotics", "Breakthrough", "Startup", "Handles", "Objects", "Like", "Humans"],
  "lock_text": true,
  "must_keep_keywords": ["Robotics Breakthrough", "Handles Objects"]
}

Fields

• hero — 2–4 words that get Bolt Classic treatment (top-left, yellow, large)
• trail — remaining words; TikTok karaoke reveal
• hero_split_at — word count of hero. Set at write time. Never null.
• hero_position — "start" (default) or "end"
• template — TextCard | SplitScreenAvatar | ImageKaraoke
• source_image — path to user-provided image (for image-led scenes)
• highlight_words — words to highlight via OCR matching (ImageKaraoke mode)
• vo_line — spoken line for this scene
• lock_text — if true, VO must preserve image text (light grammar cleanup only)

hero_position

Same layout always: hero top-left (Bolt Classic), trail karaoke below. Only controls which words are hero and when they reveal.

Value	Hero words	Trail words	Timing
"start"	First N words	Remaining	Hero shows immediately at scene start
"end"	Last N words	All preceding	Trail karaoke plays first; hero reveals when spoken

---

Pipeline Steps

Step 1 — Script

Write projects/<id>/script/video-<N>-<slug>.json with:

• id, title, duration_target_seconds, voiceover (full script text), scenes[]
• Each scene: scene_id, template, and template-specific fields

Image-grounded Script Rule (mandatory when user provides images)

When image assets are provided:

1. At least 3 scenes must preserve image text in VO (light grammar cleanup allowed).
2. Preserve key terms exactly (company names, product names, numbers, claims).
3. The VO word order must follow image text reading order.
4. Add highlight_words array to each text-image scene.
5. Include source_image mapping so scene-to-image is deterministic.
6. For scenes with no OCR text (physical shots), leave highlight_words: [] — template auto-falls back to TikTok captions.

Image-text Script Lock (REQUIRED)

If image text exists, script writing is constrained by source text:

• vo_line must stay semantically equivalent to the source image text.
• must_keep_keywords must survive unchanged in spoken line + captions/highlights.
• Do not paraphrase away numbers, named entities, model names, or claims.
• If conflict exists between style and fidelity, choose fidelity.

Step 2 — VO Generation

python3 tools/generate_vo.py \
  --script "Full script text as one string" \
  --out projects/<id>/assets/audio/voiceover-full-v001.mp3 \
  --config voices/bolt-siw1-config.json \
  --trim

• Reads pronunciation corrections from voices/bolt-siw1-config.json
• Uses locked settings: stability=0.18, similarity=0.82, style=0.72
• Trims silences (max 450ms → target 160ms)
• ⚠️ Send clip to user for approval before continuing

Step 3 — Whisper Canon

source .venv-faster-whisper/bin/activate
# Run faster-whisper on approved VO
# Per scene output: projects/<id>/script/<scene-id>.v001.words.json
# Format: { scene_id, scene_start_seconds, words: [{word, start_ms, end_ms, global_start_ms, global_end_ms}] }

Step 3b — Word Boundary QA ⚠️ REQUIRED before Step 4

After Whisper, verify every scene boundary:

for each scene:
    print(f"{scene_id} FIRST: {words[0]['word']} @ global {words[0]['global_start_ms']}ms")
    print(f"{scene_id} LAST:  {words[-1]['word']} @ global {words[-1]['global_end_ms']}ms")

Fix if wrong:

1. Move the word between scene word JSON files
2. Rebase start_ms/end_ms to new scene's local time
3. Update start_seconds/end_seconds in props
4. Regenerate avatar for any SplitScreenAvatar scene whose timing changed

Common Whisper artifacts to fix:

• skills → scales, Open Claw → OpenClaw, -in as separate token

Step 4 — Background Images / User Images

If user provided images: Copy to projects/<id>/assets/images/src/img-0N.jpg and to remotion/public/assets/backgrounds/.

If AI-generated:

python3 tools/generate_images.py \
  --script projects/<id>/script/video-<N>-<slug>.json \
  --project <id>

Step 4b — OCR + Karaoke Plan (for image-led scenes)

For every scene that uses a text-heavy image (highlight_words is non-empty):

# 1. OCR extraction
python3 tools/extract_ocr_layout.py \
  --image projects/<id>/assets/images/src/img-02.jpg \
  --out projects/<id>/analysis/scene-002.ocr.json

# 2. Karaoke plan generation (maps Whisper words → OCR bboxes)

The karaoke plan maps each Whisper word to its OCR bounding box for synchronized highlighting. Generate programmatically by fuzzy-matching normalized Whisper tokens against normalized OCR tokens.

For repeatable layover workflows, use:

python3 tools/generate_word_layover_plan.py \
  --ocr projects/<id>/analysis/<scene-id>.ocr.json \
  --words projects/<id>/script/<scene-id>.v001.words.json \
  --out projects/<id>/script/<scene-id>.v001.karaoke-plan.json \
  --preset orange_layover_archivo_black \
  --terms "survival,computer"

Output: projects/<id>/script/<scene-id>.v002.karaoke-plan.json

{
  "scene_id": "scene_001",
  "image_width": 1080,
  "image_height": 1920,
  "karaoke_words": [
    {
      "word": "Mumbai",
      "start_ms": 0,
      "end_ms": 300,
      "highlight": true,
      "bbox": {"x": 100, "y": 200, "w": 150, "h": 40}
    }
  ]
}

Attach in props as karaoke_plan on the scene object.

Step 5 — Avatar Clips (SplitScreenAvatar scenes only)

python3 tools/generate_avatar.py \
  --full-vo projects/<id>/assets/audio/voiceover-full-v001.mp3 \
  --start <scene_start_seconds> \
  --duration <scene_duration_seconds> \
  --out remotion/public/avatar/<project>-<scene-id>.mp4

Step 6 — Props Assembly

Create remotion/src/props/short<N>-v001-props.ts.

Key patterns:

• Import word JSONs and karaoke plans
• wt() helper to map word timestamps
• For ImageKaraoke scenes: set karaoke_plan on scene object
• For SplitScreenAvatar + highlights: set both avatar and karaoke_plan
• For no-text scenes: omit karaoke_plan — template auto-falls back to TikTok captions

Copy VO to remotion/public/assets/audio/. Register composition in remotion/src/compositions/registry.tsx.

Scene Timing Rules

• start_seconds = first word's global_start_ms / 1000
• end_seconds = last word's global_end_ms / 1000 — no extra buffer
• Scenes must NOT overlap: scene[n].end_seconds < scene[n+1].start_seconds

Step 7 — Render

cd remotion
npx remotion render src/index.ts short-<N>-v001 ../output/short-<N>/v001/render.mp4

Step 8 — QA Gates

• Word boundary check passed (first/last word per scene)
• No scene overlap
• OCR highlights land on correct words (spot-check 2-3 words per scene)
• Karaoke timing syncs with VO (no early/late highlights)
• Avatar face visible in frame (SplitScreen scenes)
• TikTok caption fallback works for no-text scenes
• BGM audible but not overpowering VO
• Output 1080×1920 confirmed

Step 9 — Commit

git add -A && git commit -m "short-<N> vXXX: <description>"

Output contract: output/<project>/v###/render.mp4

---

Tools Reference

Tool	Command	Purpose
generate_vo.py	python3 tools/generate_vo.py --script "..." --out <path> --config voices/bolt-siw1-config.json --trim	ElevenLabs VO generation
generate_avatar.py	python3 tools/generate_avatar.py --full-vo <path> --start <s> --duration <s> --out <path>	HeyGen avatar clip
generate_images.py	python3 tools/generate_images.py --script <path> --project <id> [--overrides <path>] [--scene <id>]	Replicate background images
extract_ocr_layout.py	python3 tools/extract_ocr_layout.py --image <path> --out <path>	Tesseract OCR word boxes
generate_highlight_plan.py	python3 tools/generate_highlight_plan.py --ocr <ocr.json> --phrases "a|b" --out <plan.json>	Phrase-to-box highlight plan
generate_word_layover_plan.py	python3 tools/generate_word_layover_plan.py --ocr <ocr.json> --words <scene.words.json> --out <scene.karaoke-plan.json> --preset orange_layover_archivo_black --terms "word1,word2"	Deterministic per-word OCR layover plan

---

Voice Config (locked)

File: voices/bolt-siw1-config.json

• Voice ID: siw1N9V8LmYeEWKyWBxv
• Stability: 0.18, Similarity: 0.82, Style: 0.72
• Pacing: max_silence 450ms, gap 160ms, trim 220ms
• Pronunciation corrections accumulate here for all future videos

---

BGM Layer

• Default: remotion/public/assets/bgm/bgm-cinematic-tech-ambient.mp3
• Configurable via script.bgm and script.bgm_volume (default 0.18)
• Fades in 1s at start, fades out 1s before end

---

HeyGen Avatar Config (locked)

• Avatar ID: a12a4b7f6e4d4c91bd9872aed715ce80
• Style: normal
• Background: #0a0a0a
• API key: HEYGEN_API_KEY in .env
• Output always re-encoded to H.264 High + yuvj420p before Remotion

---

Background Image Config

• Model: flux-dev via Replicate
• Version: 6e4a938f85952bdabcc15aa329178c4d681c52bf25a0342403287dc26944661d
• Steps: 28, guidance: 3.5, size: 768×1344 (9:16)
• API key: REPLICATE_API_TOKEN in .env

---

BackgroundZone — Shared Background Primitive

File: remotion/src/components/BackgroundZone.tsx

background: {
  type: 'screenshot',
  src: 'assets/backgrounds/my-image.jpg',
  bg_fit: 'cover',       // 'cover' (default) | 'contain' (full image visible)
  bg_dim: 0.4,           // overlay darkness 0–1. Default 0.4. 0 = no overlay
  bg_position: 'center', // 'top' | 'center' | 'bottom'
}

bg_fit	bg_dim	Best for
cover	0.1–0.4	AI-generated backgrounds, stock photos
contain	0	Product screenshots, UI captures, news screenshots

---

ImageKaraoke — OCR Word Highlighting

The ImageKaraoke system highlights words directly on source images as the VO speaks them. This replaces traditional subtitle tracks for image-led content.

How it works

1. OCR extracts word bounding boxes from the source image
2. Whisper extracts word-level timestamps from the VO
3. A karaoke plan maps each Whisper word to its OCR bbox
4. At render time, yellow highlight boxes animate left-to-right as each word is spoken

Where highlights render

• ImageKaraoke template: highlights render full-screen over the image
• SplitScreenAvatar template: highlights render in the top 62% zone alongside the background image. Avatar stays in bottom 38%.

Fallback behavior

When a scene has no karaoke_plan (e.g., physical product shots with no readable text), the ImageKaraoke template automatically renders TikTok-style karaoke captions instead.

Highlight style defaults

• Color: #FDE047 (yellow)
• Opacity: 0.56
• Animation: left-to-right wipe with Easing.out(Easing.quad)
• Anti-sliver threshold: 6% (prevents flash artifacts)
• Fade: highlights persist at reduced opacity (0.4) after word is spoken

---

Mandatory Usage

Bolt MUST always follow this pipeline for every video.

• Steps run in order: 1 → 2 → 3 → 3b → 4 → 4b → 5 → 6 → 7 → 8 → 9
• Step 3b (Word Boundary QA) is mandatory before avatar generation
• Step 4b (OCR + Karaoke Plan) is mandatory for image-led scenes
• All renders through RaaSComposition via registry
• Avatar regeneration rule: any scene boundary shift = regenerate that avatar

Highlight Logic Lock (REQUIRED)

• OCR extraction must be deterministic artifact (analysis/*.ocr.json) and committed.
• Highlight plan (*.karaoke-plan.json) must map canonical Whisper words to OCR boxes.
• Highlight reveal timing must use canonical start_ms only.
• Highlights render behind text and must not reduce text legibility.
• If OCR coverage is insufficient, fallback caption mode is required (no fake boxes).

Tool Calling Lock (REQUIRED)

• VO: use tools/generate_vo.py only.
• Avatar: use tools/generate_avatar.py only.
• Image generation: use tools/generate_images.py only.
• OCR layout: use tools/extract_ocr_layout.py only.
• Highlight plan: use tools/generate_word_layover_plan.py for per-word layover workflows (or tools/generate_highlight_plan.py for phrase-level plans).
• QA lock: overlay label text must equal OCR word and timing must equal canonical word timing.
• No raw provider API calls in production pipeline.