vlog-visual-engine-prototype

name

vlog-visual-engine-prototype

description

Prototype a local visual-rendering lane for vlog-pipeline using HTML, Canvas, Remotion, Motion Canvas, Phaser, PixiJS, or a small in-repo engine, then pair it with existing MiniMax narration/music and FFmpeg composition. Use when the user asks to explore public video-generation repos, pixel-art or animation styles, HTML-to-video rendering, local rendering engines, knowledge-point audio/card episodes, resource allocation with MiniMax only, or alternatives to Dreamina/Seedance for experimental short-video visuals.

Vlog Visual Engine Prototype

Overview

Use this skill for experimental visual production only. Keep the stable daily lanes in vlog-round-launch and vlog-daily-check unchanged unless the user explicitly asks to promote a prototype into production.

The current repo already uses MiniMax for M2.7 script generation, TTS fallback, image-01, music, and visual review, and uses FFmpeg heavily in app/mirofish/driver.py. Treat MiniMax as the narration/music/QA layer and the local visual engine as the controllable picture layer.

Knowledge Packet First

Start by giving the renderer and writer enough factual material. Do not ask MiniMax to invent facts while also writing the final episode.

For textbook-style history topics, prefer existing app/stories/* fields such as title, era, source, story body, and knowledge_points as the first source of truth.

For wild-history topics or topics without knowledge_points, make a short knowledge_packet.json before writing narration:

{
  "title": "",
  "era": "",
  "facts": [{"claim": "", "importance": 1, "confidence": "high", "source_hint": ""}],
  "timeline": [{"time": "", "event": ""}],
  "characters": [{"name": "", "role": ""}],
  "conflicts": [],
  "visual_constraints": []
}

Use MiniMax M2.7 for knowledge distillation, outline repair, script writing, and judging, but keep the factual packet separate from the final narration. This lets a later renderer create cards, timelines, maps, captions, or scenes without re-asking for background knowledge.

MiniMax Resource Budget

Use MiniMax where it adds the most value:

• M2.7: knowledge packet, outline, narration draft, emotional beats, judge pass.
• TTS: final narration only, at natural speed, usually one pass.
• Music: optional bed after the narration duration is known.
• Image-01: optional key stills, covers, or hard-to-draw scenes.
• Video generation: optional and scarce; reserve it for a few high-value cinematic inserts, not for every scene.

Use local CPU/FFmpeg/Pillow/HTML/Remotion/Motion Canvas/PixiJS/Phaser for deterministic visuals, subtitles, maps, timelines, cards, and stitching.

Default Architecture

Prototype this shape first:

1. manifest.json, one episode record, or a knowledge_packet.json supplies factual beats.
2. MiniMax M2.7 or local code turns those beats into narration text plus a renderer-agnostic cue sheet.
3. Existing MiniMax/Edge TTS creates narration audio at natural speed with max_duration=None.
4. Probe the real narration duration with ffprobe or the repo get_duration() helper.
5. A local renderer creates a 9:16 silent visual MP4 whose duration is derived from the narration, not the other way around.
6. FFmpeg combines video, narration, subtitles, and music into a proof.

Do not speed up, slow down, or atempo the narration to satisfy a fixed video duration unless the user explicitly asks for compressed timing. The voiceover carries pacing and emotion; visuals must adapt to it.

Write prototypes under app/data/visual_engine_prototypes/<date>/<engine>/ unless the user names another location.

Output Mode Decision

Choose the cheapest output that carries the idea:

• Knowledge-card audio episode: default for dense facts, exam points, chronology, maps, names, or comparisons. Render cards/timeline/captions locally and mux with narration.
• Pixel or animated local scene: use when the story has characters, conflict, or spatial action that benefits from staging.
• Hybrid episode: mostly knowledge cards plus a few generated stills or short cinematic inserts.
• Audio-only proof: acceptable for testing voice, pacing, and script before spending visual budget.

Even when the content is "not really video", prefer exporting a lightweight MP4 for platform compatibility: 9:16 cards, captions, progress, and waveform or subtle motion are enough.

Voice Defaults

Local evidence points to MiniMax audiobook_male_1 as the main tested voice family. Prefer:

• minimax:audiobook_male_1:neutral:0 for steady history narration.
• minimax:audiobook_male_1:sad:-2 for lower, darker, documentary narration.
• minimax:narrator_male_1:calm:0 as an alternate narrator voice.

Check app/dashboard/static/voice_samples/ and app/data/tonguetwister/ when the user mentions a preferred local voice. Do not claim a final preference unless it is obvious from the current task or the user confirms it.

Engine Choice

Prefer the smallest engine that can prove the visual idea:

• HyperFrames: first candidate for HTML-native knowledge cards, captions, explainers, data/timeline motion, and agent-written short videos. Use only in prototype dirs first; it requires Node.js 22+ and FFmpeg.
• Remotion: default for React/HTML/CSS/Canvas/WebGL to MP4, especially timeline-heavy shorts.
• Motion Canvas: use for code-driven explainer animation or voiceover-synchronized motion.
• Phaser or PixiJS: use for pixel-art, 2D character staging, sprite maps, or game-like scenes.
• Plain HTML plus Playwright/Puppeteer frame capture: use for a tiny proof when a full framework is too much.
• Existing Pillow/FFmpeg path in app/mirofish/driver.py: use when static cards, Ken Burns, subtitles, and narration are enough.

Before choosing a public repo, re-check current project status from primary sources because stars, maintenance, licenses, and APIs drift.

When the task is to evaluate community skills or existing agent workflows, read references/community-video-skills.md first and update it only after re-checking the linked primary sources.

Community-Informed Pattern

Borrow the production pattern, not just the library:

1. Generate the final narration first at natural speed.
2. Probe audio duration and, when possible, derive a cue sheet from transcript, manual beats, captions, or audio analysis.
3. Render visuals from a structured timeline such as [{start, end, label, visual, text}].
4. Keep renderer output deterministic: fixed resolution, fps, seeds, fonts, and asset paths.
5. Use FFmpeg only for composition, not for changing narration tempo.

Good public patterns to reuse:

• Remotion: use dynamic metadata to set durationInFrames from audio or timeline data; use audio visualization APIs for audiograms or music-reactive visuals.
• Motion Canvas: use an audio track plus time events (waitUntil, useDuration) for voiceover-synced animation.
• JSON timeline engines such as Editly or VideoFlow: useful when agents should emit portable structured video specs instead of hand-written React.
• Existing repo Python/FFmpeg path: useful for quick proofs and production-friendly fallback renders.

For this repo, prefer a renderer-agnostic intermediate timeline before choosing a framework. That keeps manifest -> narration -> timeline -> renderer -> ffmpeg portable.

Public Repo Scan

For any candidate, inspect and report:

• license and whether commercial video output is safe enough for experiments
• current maintenance activity and latest release/commit
• Windows/WSL/Docker viability
• deterministic rendering or frame seeking support
• 9:16 output support
• Chinese font handling
• FFmpeg export path
• whether it can run without cloud accounts or browser login

Do not install or vendor a public repo into production paths on the first pass. Clone or scaffold prototypes in the prototype directory, then summarize the promotion path separately.

Minimum Proof

A good first prototype is intentionally small:

• one episode or one scene, usually 10-15 seconds after natural narration
• 1080x1920 MP4
• MiniMax or existing TTS narration generated with natural pacing
• optional background music
• subtitles if the existing subtitle path is cheap to reuse
• no upload queue changes
• no daily VPS generation changes

Validation:

• ffprobe reports 1080x1920, expected duration, and audio stream when audio is included.
• The reported video duration matches the natural narration duration within normal frame rounding.
• The generated MP4 opens locally or can be inspected through screenshots.
• The summary names the engine, source manifest/episode, output path, and what would be needed to make it production-ready.

Promotion Rule

Only promote a prototype after one full proof looks better or cheaper than the existing Dreamina/Seedance lane for that style. Promotion should be a separate change with config, dashboard status, retry behavior, and queue handling reviewed explicitly.