whiteboard-speak

name

whiteboard-speak

description

Create a complete AI whiteboard infographic explainer video (any duration) from a topic, viewpoint, or rough script. Use when the user asks for a whiteboard video, AI whiteboard explainer, 白板信息图讲解视频, or explicitly asks to use whiteboard-speak; produce editable HyperFrames, preview.mp4, action/camera QA, keyframes, and integration_report.md with automatic PNG providers when configured and a safe interactive fallback.

Whiteboard Speak

Turn the user's topic or rough script into a reviewable whiteboard-speak project. Treat this Skill as the only public entrypoint. Use B, C, Creator, D, E, and the orchestrator from the bundled runtime/ as internal implementation modules.

Resolve the installation

1. Treat the directory containing this SKILL.md as SKILL_ROOT.
2. Read SKILL_ROOT/installation.json when present.
3. Resolve the bundled runtime as SKILL_ROOT/runtime/ for an installed copy. In a source checkout, resolve the repository root two directories above SKILL_ROOT.
4. Run the deterministic environment check before the first video in a session:

python3 <SKILL_ROOT>/scripts/doctor.py --json --output-dir <OUTPUT_PARENT>

Report the install, render, output, and image statuses separately. Do not treat interactive image handoff as an installation failure.

Start from natural language

Accept a topic, viewpoint, or rough script directly. Do not require the user to create an input file. Support any video duration — follow the user's requested length, or use the content itself to determine the natural duration.

Default the output parent to the user's current working directory, not the managed Skill installation. Create:

<current-working-directory>/whiteboard-runs/YYYYMMDD-HHMMSS-<topic-slug>/

Write the original request to topic_input.txt inside that project directory.

Use a lowercase ASCII topic-slug with letters, digits, and hyphens. Fall back to whiteboard-speak when the topic cannot be represented safely. If the path already exists, append -2, -3, and so on; never overwrite an earlier run implicitly.

Execute the internal pipeline

Read these bundled internal instructions before executing their stage. Installed packages name each internal entry INTERNAL_SKILL.md so Agent discovery exposes only whiteboard-speak; source checkouts use SKILL.md in the same module directory.

1. runtime/ip-cognition-script-polisher/INTERNAL_SKILL.md
2. runtime/ip-hand-drawn-infographic-planner/INTERNAL_SKILL.md
3. runtime/hand-drawn-infographic-creator/INTERNAL_SKILL.md
4. runtime/hand-drawn-infographic-video-board/INTERNAL_SKILL.md
5. runtime/whiteboard-infographic-video-renderer/INTERNAL_SKILL.md
6. runtime/whiteboard-infographic-pipeline-orchestrator/INTERNAL_SKILL.md
7. runtime/whiteboard-infographic-pipeline-orchestrator/references/runbook.md
8. runtime/whiteboard-infographic-pipeline-orchestrator/references/contracts.md

Run the fixed sequence:

1. Preserve the user's stance and create the six-part B script package.
2. Create semantic C board plans without bbox, camera, cursor, or animation geometry.
3. Create final hand-drawn infographic prompts and review notes.
4. Route every required board prompt through the configured image provider.
5. Complete automatic PNG validation or the interactive image handoff described below.
6. Write and validate board_asset_manifest.json.
7. Calibrate generated-image bboxes when needed. --provider auto tries the agent backend (Claude vision via ANTHROPIC_AUTH_TOKEN) when no OPENAI_API_KEY is configured, falling back to VLM, OCR, or mock as available.
8. Generate D into board_source_for_e/.
9. Ask the user to choose a player format, then render E accordingly.
10. Run asset identity and action/camera QA, inspect keyframes, and write integration_report.md.
11. Run validate_release_candidate.py; do not call the video complete unless it writes PASS acceptance reports.

Prefer the smallest board count that communicates the idea. For a typical short video (30-60s) 1-2 boards is usually enough; scale up for longer content as needed. Treat bundled examples as test fixtures, not board-count requirements.

Player format choice

Before rendering (step 9), ask the user which player format they want. Explain the trade-offs briefly, then proceed with their choice.

Format	说明	优点	限制
HyperFrames（默认）	标准单页播放器，CDN 加载 GSAP，固定 1920×1080	原生支持 HyperFrames CLI 导出 MP4，画质无损	需联网加载 GSAP，窗口不自适应
独立网页	自包含 HTML，本地 GSAP，自适应窗口，多板自动切换	离线可用，双击即看，适合预览和分享	不能直接导出 MP4，需额外步骤

独立网页导出 MP4

独立网页也能出视频，两条路：

1. 浏览器 + FFmpeg 渲染（推荐）：用 headless Chrome 逐帧截图 → FFmpeg 合成 MP4，画质接近无损，跟 HyperFrames 导出效果相当
2. 直接录屏：OBS / 系统录屏，最简单但画质有损——受屏幕分辨率、编码参数、系统性能影响，不如渲染方案清晰

用户没明确说要 MP4 时，不用主动推录屏方案。如果用户选了独立网页但后面又要视频，优先走浏览器渲染。录屏作为兜底。

• 用户说"做个视频"、"要 MP4" → 选 HyperFrames
• 用户说"做个网页"、"预览看看"、"分享个链接" → 选独立网页
• 用户没明确说 → 默认 HyperFrames，但简单提一句可选独立网页

Use scripts from the bundled runtime with absolute paths derived from SKILL_ROOT; never assume the current directory is the Git clone or the Skill installation.

In source-checkout testing only, replace INTERNAL_SKILL.md with the module's original SKILL.md.

Route image generation

After C and Creator prompts exist, run:

python3 <RUNTIME>/whiteboard-infographic-pipeline-orchestrator/scripts/generate_board_images.py \
  --project-dir <PROJECT> \
  --provider auto

Interpret the exit status:

• 0: all PNGs passed validation and board_asset_manifest.json was written; continue to calibration and D/E.
• 3: no automatic provider is configured or some interactive PNGs are missing; generate previews with the available image tool and follow every exact path in image_generation_report.json.
• 2: provider, API, PNG validation, or manifest generation failed; stop and report the error without claiming image success.

auto only uses OpenAI when WHITEBOARD_IMAGE_PROVIDER=openai is explicitly configured. The mere presence of OPENAI_API_KEY must not trigger a billable request. Before the first billable call, state the provider, model, size, and quality being used. Never print, persist, or expose the API key.

For custom providers, use WHITEBOARD_IMAGE_PROVIDER=command and an executable WHITEBOARD_IMAGE_COMMAND; read references/image-providers.md for its argument contract.

Keep the interactive fallback honest

Interactive remains the safe default. If the image tool returns preview images without a stable local path:

• Stop once all preview images are generated.
• Ask the user to save each preview as <project>/images/<boardId>.model-generated.png.
• List every exact required path.
• Resume only after the files exist and pass PNG validation.
• Never search hidden caches, invent URLs, reuse old assets, or substitute D SVG previews or placeholders.

Do not claim zero-human automation unless image_generation_report.json has status=complete, automatic=true, every board is generated or reused, and the manifest records previewChecked=false.

Require the product outputs

Do not call a run complete until it contains and validates at least:

video/preview.mp4
video/hyperframes/
video/keyframes/
video/renderer_report.json
image_generation_report.json
sync/action_timing.json
sync/camera_plan.json
sync/action_camera_qa_report.md
sync/action_camera_qa_report.json
integration_report.md
v1_release_acceptance.json
v1_release_acceptance.md

Also require the B/C packages, local model-generated PNG manifest, D control package, measured narration timing, captions, HyperFrames lint/validate/inspect results, and asset identity check defined by the internal contracts.

Record all PASS/WARN/FAIL results and the manual image source in integration_report.md. A renderer warning from a deliberately skipped render is not equivalent to real-video acceptance.

Subtitle quality standards

Generated subtitles must follow these rules (剪映/CapCut conventions):

1. No punctuation marks — strip all 。！？，、——；：""''「」『』【】（）《》…— from displayed text
2. 8-15 characters per line — split long captions at sentence boundaries (。！？), then at comma/pause boundaries (，、——；) for sub-15-char chunks
3. Bottom-center positioning — bottom: 60px, flexbox-centered, no background bar or gradient shade
4. High-contrast text — white #fff, 6px black stroke at 75% opacity, 42px font, weight 700
5. Clean transitions — instant cut (opacity set at segment start, quick fade-out at segment end), no Y-translation or slide animation
6. Timing sync — caption visibility tied to segment.start / segment.end from measured audio timing. VTT cue data should be embedded when sub-phrase sync is needed.
7. Annotation completeness — every action must include tl.set(node, {opacity:1}) and tl.to(node, {strokeDashoffset:0}) so hand-drawn marks are visible

Renderer timing contract

The renderer must write data.js MOTION_PLAN segment boundaries that match voiceover_timing.json exactly. Do not recalculate or approximate — copy start, speechEnd, and end verbatim from the measured timing. Mismatched boundaries cause subtitle desync and missing phrases.

Audio duration gate (REQUIRED)

Before writing MOTION_PLAN, verify audio duration matches the calculated total:

ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 audio/narration.wav

• Compare against MOTION_PLAN.composition.duration
• Must match within 0.1s tolerance
• If mismatch > 0.1s, the audio processing pipeline (loudnorm, concat, etc.) changed the duration — regenerate or recalculate timing
• Do not proceed until durations match

Data validation gate (REQUIRED)

Before rendering, validate the entire project:

node whiteboard-infographic-video-renderer/scripts/validate_project.mjs whiteboard-runs/<run-id>/

The script checks:

• audio duration vs MOTION_PLAN.composition.duration
• All board SVGs exist
• All element IDs are unique
• Every action references a real element and annotation
• Every anchor text exists in the segment's caption
• SVG files have no stray annotation elements
• Annotation definitions are complete (underline needs start/end, circle needs center/radius, etc.)

Do not render if validation fails. All FAIL items must be resolved first.

Action timing (anchor — recommended)

Actions use anchor text to align with subtitle timing automatically:

{"type":"box","element":"draw_step3","anchor":"手绘描边","duration":0.7}

The renderer:

1. Runs splitCaption() on the segment's VTT text to get all subtitle phrases
2. Finds the FIRST phrase containing the anchor text
3. Calculates the exact time from character position within that VTT cue
4. Triggers the action at phraseTime + 0.3s

Matching rules:

• Scoped to current segment only — no cross-segment matching
• First match wins (deterministic)
• If no match → console.warn + fires at segment midpoint
• If multiple matches → console.warn + uses first match
• Anchor should be a unique substring (e.g. "手绘描边" not "AI")

Fallback chain: anchor → phraseIndex (compatible) → offset (deprecated)

Annotation coordinate rules

Annotations must NOT overlap SVG static content:

• underline: y-coordinate = target baseline + 30px minimum offset below
• box: padding 10px around target bounds (slightly larger, not overlapping edges)
• circle: center should encompass text with 10-20px margin
• check: drawn in clean space, not on existing SVG graphics

SVG files are static backgrounds. Do NOT embed cursor pointers, annotation circles, or highlight marks in SVG files — these belong in the GSAP annotation layer only.

Self-check before delivery

Every whiteboard-speak output must pass this checklist:

• audio duration == MOTION_PLAN composition.duration (≤0.1s tolerance)
• SVG files contain NO static annotation elements (no cursor, circle, box, or tick marks)
• All board SVGs updated in assets/board/ match source boards/
• Action offsets verified against subtitle character timing
• Annotation coordinates do not overlap SVG content (underline 30px below, boxes 10px padded)
• Every board plays through without JS console errors
• Subtitle text contains NO punctuation marks

Finish with:

python3 <RUNTIME>/whiteboard-infographic-pipeline-orchestrator/scripts/validate_release_candidate.py \
  --project-dir <PROJECT>

Additional reference

Read references/install-layout.md when installation resolution or target-agent behavior is unclear.

Read references/image-providers.md before configuring OpenAI or a custom command provider.