| name | workflow-ai-generation |
| description | Generate media from scratch with 2026 open-source AI — TTS voiceover (Kokoro / OpenVoice / Piper), image gen (FLUX-schnell / Kolors / Sana / ComfyUI), video gen (LTX-Video / CogVideoX / Mochi / Wan), music (Riffusion / YuE), lipsync talking heads (LivePortrait / LatentSync), OCR (PaddleOCR / Tesseract 5 / TrOCR), zero-shot tagging (CLIP / SigLIP / BLIP-2 / LLaVA). Strict commercial-safe license filter. Use when the user says "generate a video", "TTS voiceover", "AI explainer video", "clone my voice", "generate music", "AI image", "digital human", or anything about from-scratch AI media. |
| argument-hint | ["prompt"] |
Workflow — AI Generation
What: Synthesize new media with open-source, commercial-safe AI models. Strict license filter: Apache-2 / MIT / BSD / GPL. NC / research-only models are always-dropped.
Skills used
media-tts-ai, media-whisper, media-sd, media-svd, media-musicgen, media-lipsync, media-depth, media-ocr-ai, media-tag, media-demucs, media-denoise-ai.
Tool matrix
TTS (media-tts-ai)
| Model | License | Best for |
|---|
| Kokoro | Apache-2 | general default |
| OpenVoice | MIT | voice cloning |
| CosyVoice | Apache-2 | Chinese |
| Chatterbox | MIT | expressive |
| Piper | MIT | embedded / offline |
| StyleTTS2 | MIT | single-voice quality |
| Parler | Apache-2 | style prompting |
| Bark | MIT | creative / SFX |
| Orpheus | Apache-2 | modern expressive |
DROPPED: XTTS-v2 (CPML NC), F5-TTS (research).
Image (media-sd)
| Model | License | Best for |
|---|
| FLUX-schnell | Apache-2 | default (4-step distilled) |
| Kolors | Apache-2 | bilingual EN/ZH |
| Sana | Apache-2 | fast 4K |
| ComfyUI | GPL-3 | node-graph workflows |
DROPPED: FLUX-dev (NC), SDXL / SD3 base (restrictive).
Video (media-svd)
| Model | License | Best for |
|---|
| LTX-Video | Apache-2 | fastest |
| CogVideoX | Apache-2 | high quality (slower) |
| Mochi | Apache-2 | cinematic |
| Wan | Apache-2 | versatile |
DROPPED: Stable Video Diffusion (NC research).
Music / SFX (media-musicgen)
| Model | License | Best for |
|---|
| Riffusion | Apache-2 | spectrogram-diffusion |
| YuE | Apache-2 | long-form structured |
DROPPED: Meta MusicGen (CC-BY-NC).
Lipsync (media-lipsync)
| Model | License |
|---|
| LivePortrait | MIT |
| LatentSync | Apache-2 |
DROPPED: Wav2Lip (research), SadTalker (NC).
OCR (media-ocr-ai)
| Model | License | Best for |
|---|
| PaddleOCR | Apache-2 | Latin + Chinese default |
| Tesseract 5 | Apache-2 | fastest classical |
| TrOCR | MIT | handwriting |
| EasyOCR | Apache-2 | multilingual |
DROPPED: Surya (commercial restriction).
Tagging / captioning (media-tag)
CLIP (MIT), SigLIP (Apache-2), BLIP-2 (BSD), LLaVA (Apache-2 but needs Llama-2 backbone — check license).
Stem separation (media-demucs)
htdemucs 4-stem, htdemucs_6s 6-stem (adds guitar + piano).
Speech-to-text (media-whisper)
whisper.cpp (MIT fastest CPU), faster-whisper (MIT CUDA-accelerated).
Example composite workflows
Explainer video from script
script → TTS voiceover (media-tts-ai Kokoro) → slide images (media-sd FLUX-schnell) → B-roll clips (media-svd LTX-Video) → host animation (media-lipsync LivePortrait) → background music (media-musicgen Riffusion) → assemble (ffmpeg-cut-concat) → mix (ffmpeg-audio-filter + sidechain ducking) → loudness normalize (media-ffmpeg-normalize) → auto-burn subtitles (media-whisper + ffmpeg-subtitles) → transcode H.264 → YouTube upload (media-cloud-upload).
Multilingual voice clone
30 s clean reference → DeepFilterNet denoise → OpenVoice clone to EN/ES/FR/JP → per-track denoise → mux with localized subs.
Book cover + trailer
FLUX-schnell cover variations → LLaVA auto-describe for alt-text → Mochi cinematic trailer → StyleTTS2 narrator → YuE cinematic score → assemble.
Automated podcast chapter thumbnails
scenedetect chapter boundaries → extract frame per chapter → BLIP-2 caption → Kolors generate thumbnail from caption → embed chapter metadata.
Digital human with cloned voice
OpenVoice clone → LivePortrait drives portrait → FLUX-schnell branded background → chromakey + RVM matte composite → Riffusion audio bed → transcode + upload.
Gotchas
- License discipline. Always check the skill's
references/LICENSES.md BEFORE adopting a new model. HuggingFace license field is NOT authoritative. Pin model weights to specific commit hashes.
- GPL-3 in ComfyUI / RVM — requires source distribution if modified/redistributed. Use dynamically / in pipeline, not bundled into proprietary product.
- All Layer 9 skills need GPU for reasonable throughput (10–50× slower on CPU).
- TTS sample rates vary: Kokoro 24 kHz, Piper 22.05 kHz, StyleTTS2 24 kHz, OpenVoice 24 kHz. Resample ALL to 48 kHz before mixing.
- Voice cloning needs CLEAN reference. Noisy sample → noisy clone. DeepFilterNet the reference first.
- FLUX-schnell is 4-step distilled. Do NOT push
--steps above 4–8 — wastes compute, no quality gain.
- ComfyUI workflow JSONs pin specific node versions. Use ComfyUI Manager to install matching nodes.
- LTX-Video prompt adherence is phrasing-sensitive. "cat running" ≠ "running cat".
- CogVideoX-5b needs ~20 GB VRAM. 2b variant runs on 8 GB at lower quality.
- Riffusion produces 5.11-second clips natively. Chain with crossfade for longer, or use YuE for structured long-form.
- LivePortrait expects clean frontal portrait. Angled faces, glasses, occluded mouths degrade output.
- LivePortrait outputs 512×512 by default. Upscale with
media-upscale for larger.
- Whisper
large-v3 is 3 GB. Test with base.en (140 MB) first — quality gap to medium (1.5 GB) is small for clean audio.
- Whisper hallucinates on silence. Trim leading/trailing with
silenceremove.
- Whisper word-level timestamps require
--word_timestamps True (faster-whisper) or --max-len 1 --split-on-word (whisper.cpp).
- Whisper
--language auto is fragile on accented speech. Specify language explicitly.
- Diarization is external to Whisper. Use pyannote.audio (MIT) or simple-diarizer.
- OCR selection: PaddleOCR for Latin+Chinese, TrOCR for handwriting, Tesseract for speed on clean docs.
- CLIP / SigLIP / BLIP-2 need fixed-resolution inputs (224/336/384/448). The
tagctl.py script resizes internally.
- LLaVA needs an LLM backbone (Vicuna / Llama-2 7B+). Llama-2 community license has revenue caps.
Related
workflow-ai-enhancement — for enhancing EXISTING footage (not generating new).workflow-podcast-pipeline — for podcast-specific AI workflows.workflow-analysis-quality — VMAF + QC on AI-generated output.
