ai-podcast

## AI Podcast Generator Create multi-person talking head podcast videos using the inference.sh pipeline: portrait generation → TTS audio → avatar video → merge. Supports real humans (via Phota), 3D mascots, illustrated characters, and mixed casts. Use when the user wants to create a podcast, talking head video, demo reel, promotional conversation, or any multi-speaker video content. ### Pipeline Overview ``` Characters (images) → TTS (audio per turn) → Avatar (video per turn) → Merge (final video) ``` ### Process #### Step 1: Character Creation Choose the right tool per character type: | Character Type | Tool | Notes | |---------------|------|-------| | **Real human (new)** | `pruna/p-image` | 16:9, `prompt_upsampling: true`. Quick, no training needed, but identity won't be consistent across multiple generations. | | **Real human (consistent ID)** | `phota/generate` with `[[profile_id]]` | Consistent identity across all shots. Requires a trained Phota profile first (see below). | | **Brand mascot / logo character** | `google/gemini-3-pro-image` | Pass logo + character sheet as reference images | | **Illustrated / stylized** | `google/gemini-3-pro-image` | Pass style reference as input image | **Training a Phota identity (optional but recommended for humans):** If you need a real human character with consistent identity across multiple angles and shots, train a Phota profile first: ```bash belt app run phota/train --input '{ "images": ["url1.jpg", "url2.jpg", ...], "wait": true }' --save profile.json ``` - Requires 30-50 face images of the subject - Training takes a few minutes with `wait: true` - Returns a `profile_id` you then use in `phota/generate` as `[[profile_id]]` in prompts - The profile is reusable forever — train once, generate unlimited shots If you don't need cross-shot consistency (e.g. single-speaker video, one angle only), `pruna/p-image` is simpler and cheaper. **Character sheets first, podcast frames second:** 1. Generate a character sheet (plain white background, multiple angles) for each character 2. Then place characters into the podcast studio setting using the sheet as reference **For branded characters (logo on clothing):** 1. Generate the character with a plain version of the garment 2. Use `phota/edit` with the logo as a second reference image to add the logo 3. Always pass the logo image alongside character references when generating new angles #### Step 2: Alternate Angles Generate at least 2 angles per character for visual variety: | Angle | When to use | |-------|-------------| | **Front/medium** | Establishing shots, opening, closing | | **Close-up** | Reactions, emotional moments, punchy lines | For close-ups, prompt for "tight framing, chest up, shallow depth of field" — not "turned to the side" (which just makes them look away). **Identity consistency rules:** - For real humans with a Phota profile: use `phota/generate` or `phota/edit` for new angles — Gemini does not preserve facial identity and will produce a different person - For real humans without a Phota profile: try to generate all needed angles in one go with `pruna/p-image`, or consider training a Phota profile if you need many shots - For mascots/illustrations: Gemini 3 Pro is fine, pass the established frame as reference **Framing rule:** Use tight framing on individual speakers. Wide shots with multiple seats show empty chairs when only one person is on screen. #### Step 3: QA Frames Before proceeding, visually inspect all frames for: - Extra people in the background - Multiple microphones (should be single mic per shot) - Wrong or distorted logos - Inconsistent character identity across angles - Weird artifacts (extra limbs, merged objects) Fix issues before generating video — re-rendering video is the most expensive step in the pipeline. #### Step 4: Write the Script **Rules for natural conversation:** - Write it like a real conversation, NOT like people reading ad copy in turns - Include reactions ("wait, hold on", "that is wild"), interruptions, and follow-up questions - Vary turn length — short reactions (1 sentence) mixed with longer explanations (2-3 sentences) - The host should ask real questions, not set up obvious talking points - Keep total duration target in mind: ~2.5 words/second for natural speech at 1.05x rate **Duration guide:** | Target | Words | |--------|-------| | 15s | ~38 words | | 30s | ~75 words | | 60s | ~150 words | #### Step 5: Generate TTS Audio Use `inworld/text-to-speech-2` for each turn. ```bash belt app run inworld/text-to-speech-2 --input '{ "text": "...", "voice_id": "...", "speaking_rate": 1.05, "audio_encoding": "MP3" }' --save output.json ``` **Voice selection:** - Generate samples with the same line across candidate voices BEFORE committing - Let the user listen and approve voices - Good podcast voices: Tyler, Nate, Lauren, Kelsey, Naomi, Anjali (EN_US) - Use `inworld/text-to-speech-2:voices` to list all available voices **Speaking rate:** - Default to **1.05** for natural podcast pacing - Use **1.1** for short snappy reactions - **NEVER go below 1.0** — sounds slow and disengaging - Keep rate consistent per character across all their turns All TTS turns can run in parallel (cheap, fast ~2-8s each). #### Step 6: Generate Video Clips Use `pruna/p-video-avatar` for each turn. ```bash belt app run pruna/p-video-avatar --input '{ "image": "<character_frame_url>", "audio": "<tts_audio_url>", "resolution": "720p", "video_prompt": "..." }' --save output.json ``` **Critical: Run clips SEQUENTIALLY, not in parallel.** Parallel runs hit the same GPU and cause CUDA OOM failures. Each clip takes 15-90s depending on audio length. **Angle assignment plan:** Alternate between front and close-up shots across turns for visual variety. Example for 6 turns: ``` T1: Speaker A — front T2: Speaker B — front T3: Speaker C — front (or close-up) T4: Speaker A — close-up T5: Speaker B — close-up T6: Speaker A — front ``` #### Step 7: Merge Use `infsh/media-merger` to stitch all clips into the final video. ```bash # Build input JSON { "media_files": [ {"file": "<clip1_url>"}, {"file": "<clip2_url>"}, ... ], "fps": 24, "output_format": "mp4" } belt app run infsh/media-merger --input merger_input.json --save final.json ``` Merger is free and takes 2-6 minutes depending on total duration. ### Rules 1. **Gemini does not preserve human facial identity** — generating alternate angles of a real human with Gemini will produce a different person. For identity-consistent human shots, use Phota with a trained profile_id, or generate all angles in a single batch. This was learned after Gemini produced an entirely different face for a close-up that was supposed to match the front shot. 2. **NEVER run p-video-avatar clips in parallel** — they compete for GPU memory and fail with CUDA OOM. Run them sequentially. This was learned after 2 of 3 parallel runs failed. 3. **NEVER set speaking_rate below 1.0** — it sounds artificial and disengaging. Default to 1.05. Learned from user feedback that 0.9 rate "felt weird and disengaging." 4. **ALWAYS QA frames before generating video** — video generation is the most expensive step in the pipeline. Catching a double mic or wrong logo in the image stage is cheap to fix. Catching it after video generation means re-rendering the entire clip. 5. **ALWAYS use tight framing for individual speaker shots** — wide/establishing shots show empty seats where other speakers should be. Frame from waist or chest up so no empty chairs are visible. 6. **ALWAYS pass the logo as a reference image** when generating branded characters — describing a logo in text produces wrong results. Pass the actual logo file as a second image input. 7. **ALWAYS get voice approval before full production** — generate samples with the same line across 5-8 candidate voices and let the user pick before committing to the full script. 8. **Script should read like a conversation, not an ad** — people reading ad copy in turns sounds fake. Include reactions, interruptions, varied turn lengths, and genuine questions. The host should have personality, not just set up talking points. ### App Reference | App | Purpose | |-----|---------| | `pruna/p-image` | Generate portraits from text | | `phota/train` | Train identity profile from 30-50 face images | | `phota/generate` | Generate images with trained identity via `[[profile_id]]` | | `phota/edit` | Edit images preserving identity of known subjects | | `google/gemini-3-pro-image` | Image gen/edit, mascots, style transfer | | `inworld/text-to-speech-2` | Text to speech, 100+ languages, voice steering | | `pruna/p-video-avatar` | Portrait + audio → talking head video | | `infsh/media-merger` | Concatenate video clips into one video | Use `belt task cost <task-id>` to check the cost of any individual task.