video-engine

name	video-engine
description	Video generation pipeline guide for Synthex. Documents the 4-provider architecture (Runway, Synthesia, D-ID, Remotion), FFmpeg post-processing, media library integration, and God Mode gating.
metadata	{"author":"synthex","version":"1.0","engine":"synthex-ai-agency","type":"reference-skill","triggers":["video generation","video pipeline","remotion","ffmpeg","video rendering","runway","synthesia","d-id","avatar video","video processing","media generation"]}
context	fork

Video Engine — Video Generation Pipeline Guide

Purpose

Synthex generates videos through a multi-provider pipeline with 4 providers, FFmpeg post-processing, ElevenLabs voice integration, and a Supabase-backed media library. Remotion is God Mode only.

This skill documents the full pipeline, provider differences, and the architectural decisions at each stage.

Pipeline Flow

User Request (script, prompt, image, or template)
  │
  ▼
API Route: /api/media/generate/video
  │  - APISecurityChecker (JWT auth)
  │  - Zod schema validation
  │  - God Mode gate (Remotion only)
  ▼
Provider Selection (explicit or auto-select)
  │
  ├─ Runway ML ──── text-to-video, image-to-video, motion (Gen-3)
  ├─ Synthesia ──── avatar video with TTS (scripts)
  ├─ D-ID ────────── talking head from image + script
  └─ Remotion ────── programmatic React rendering (GOD MODE)
  │
  ▼
Async Processing (all providers return video_id for polling)
  │
  ▼
Status Polling: GET /api/media/generate/video?videoId=X&provider=Y
  │
  ▼
Media Library: Supabase Storage (media_assets table)
  │
  ▼
Optional: FFmpeg post-processing (fluent-ffmpeg)
  │
  ▼
Publish to platforms (YouTube, Instagram, LinkedIn, TikTok)

Provider Matrix

Provider	Types	Auth Pattern	API Base	Access	Env Var
Runway ML	text-to-video, image-to-video, motion	Bearer token	https://api.runwayml.com/v1	All users	RUNWAY_API_KEY
Synthesia	avatar (script → video)	API key header	https://api.synthesia.io/v2	All users	SYNTHESIA_API_KEY
D-ID	avatar (image + script → talking head)	Basic auth	https://api.d-id.com	All users	DID_API_KEY
Remotion	programmatic (React compositions)	N/A (local)	N/A	God Mode	None

Auto-Selection Logic

When provider is not specified in the request:

Video Type	Default Provider
text-to-video	Runway ML
image-to-video	Runway ML
motion	Runway ML
avatar (no image)	Synthesia
avatar (with image)	D-ID

Remotion is never auto-selected — it must be explicitly requested, and only by owner accounts.

God Mode Gating

Server-Side (API Route)

God Mode gating applies to Remotion only (rendered via the admin Remotion Studio surface). The Remotion Studio page is gated at the layout level — see app/dashboard/admin/remotion-studio/page.tsx and admin/layout.tsx for the isOwnerEmail() check.

Client-Side (UI)

• Check user.isMultiBusinessOwner from useUser() hook
• Hide God Mode providers from dropdowns for non-owners

Page-Level (Remotion Studio)

• /dashboard/admin/remotion-studio protected by admin/layout.tsx
• Layout calls isOwnerEmail(user.email) and redirects non-owners

Key Files

File	Purpose
lib/services/ai/video-generation.ts	Multi-provider service (generate + status check)
app/api/media/generate/video/route.ts	Video API route (POST generate, GET status, PUT batch)
lib/services/media-library.ts	Supabase media asset storage
lib/services/ai/voice-generation.ts	ElevenLabs TTS for voiceovers
lib/remotion/Root.tsx	Remotion composition registry
lib/remotion/compositions/	React video compositions (SocialReel, ExplainerVideo)
app/dashboard/admin/remotion-studio/page.tsx	God Mode Remotion preview
app/api/admin/remotion/route.ts	Composition listing API

Database Model

model VideoGeneration {
  id                  String   @id @default(cuid())
  userId              String
  organizationId      String?
  title               String
  topic               String?
  style               String   // 'social-reel' | 'explainer' | 'how-to'
  duration            String   // '15-60s' | '2-3m' | '3-5m'
  status              String   // 'pending' | 'generating' | 'rendered' | 'published' | 'failed'
  scriptContent       Json     // { scenes, voiceover, visualDescriptions }
  videoUrl            String?
  thumbnailUrl        String?
  youtubeVideoId      String?
  scheduledPlatforms  String[] // ['youtube', 'instagram', 'linkedin']
}

FFmpeg Integration

Packages installed:

• @ffmpeg-installer/ffmpeg — FFmpeg binary
• @ffprobe-installer/ffprobe — FFprobe binary
• fluent-ffmpeg — Node.js FFmpeg API
• get-video-duration — Duration extraction

Use for: overlays, watermarks, format conversion, trimming, concatenation.

ElevenLabs Voice Integration

File: lib/services/ai/voice-generation.ts

9 premium voices available. Used for video voiceovers when scripts are provided. Supports voice cloning from audio samples.

Common Mistakes

Mistake	Why It's Wrong	Correct Pattern
Not checking provider env var	Silent failure	Each provider function checks first
Rendering Remotion server-side on Vercel	50MB limit, 60s timeout	Use client-side Player or Lambda
Not polling for status	Videos are async	All providers return video_id for polling
Skipping media library save	Assets lost	Default saveToLibrary: true

Environment Variables

Variable	Provider	Required
RUNWAY_API_KEY	Runway ML	For video generation
SYNTHESIA_API_KEY	Synthesia	For avatar videos
DID_API_KEY	D-ID	For talking heads
ELEVENLABS_API_KEY	ElevenLabs	For voice generation

Reference skill: This is a read-only architecture guide — it documents existing systems and does not generate creative or code output. No capability uplift block is needed.