heygen-translate

# HeyGen Video Translation Translate and dub an existing video into 175+ languages. The system clones the presenter's voice into the target language, re-syncs their lips to the new audio, and returns a fully dubbed video. You provide a source video and a target language — the engine handles transcription, translation, voice cloning, lip-sync, and (optionally) burned-in captions. This is **not** new-video generation. The presenter, performance, framing, and brand assets in the original video are preserved. Translation rides on top of what's already there. ## User-Facing Behavior 1. **Be concise.** Don't dump translation IDs, raw API payloads, or status JSON in chat. Report the result (video link, language) not the plumbing. 2. **No internal jargon.** Don't say "polling", "video_translation_id", "asset_id", "v3 endpoint". Say "translating", "almost done", "your file". 3. **Polling is silent.** Background long-running translations and only speak on (a) result delivered, (b) >5 min stall (one update), (c) hard failure. 4. **One result, one message.** When the video is done, send the link plus a one-line summary (target language, duration, mode). Not every API field. 5. **Don't narrate transport choice.** MCP vs CLI vs OpenClaw plugin is internal. Pick silently at session start; never mention which is in use. 6. **Communicate in the user's language.** Detect from their first message. Replies, confirmations, and questions in their language. Technical CLI/API directives stay in English. ## API Mode Detection **Pick one transport at session start. Never mix, never switch mid-session, never narrate the choice.** Detect in this order: 1. **OpenClaw plugin mode** — If running inside OpenClaw and the `video_generate` tool exposes a HeyGen translation model, prefer that. *Currently the plugin generates videos but does not expose translation directly — fall through to the next tier until HeyGen ships translation through `video_generate`.* 2. **CLI mode (API-key override)** — If `HEYGEN_API_KEY` is set in the environment AND `heygen --version` exits 0, use CLI. API-key presence is an explicit signal that the user wants direct API access. 3. **MCP mode** — No `HEYGEN_API_KEY` AND HeyGen MCP tools are visible (`mcp__heygen__*`). OAuth auth, runs against the user's plan credits. 4. **CLI mode (fallback)** — MCP tools not available AND `heygen --version` exits 0. Auth via `heygen auth login`. 5. **Neither** — tell the user once: "To use this skill, connect the HeyGen MCP server or install the HeyGen CLI: `curl -fsSL https://static.heygen.ai/cli/install.sh | bash` then `heygen auth login`." ### Auth verification (run before any API call) After mode detection, verify auth actually works before entering Phase 1. This avoids wasting the user's time gathering inputs only to hit an auth error on submit. - **MCP mode:** auth is handled by OAuth — no check needed. - **CLI mode:** run `heygen auth status` (silent). If it exits 0, proceed. If it exits non-zero (no key, expired, invalid): 1. Ask the user: *"I need your HeyGen API key to proceed. You can grab one from https://app.heygen.com/settings?nav=API — paste it here."* 2. Once they provide it, persist it: `echo "<key>" | heygen auth login` (writes to `~/.heygen/credentials`, survives across sessions). 3. Verify: `heygen auth status`. If still failing, surface the error and stop. This is a **one-time setup**. Once `heygen auth login` persists the key, future sessions pick it up automatically. Don't ask again if `heygen auth status` passes. **Hard rules:** - **Never call `curl api.heygen.com/...`** Every operation in this skill has a CLI command and (where supported) an MCP tool. Use those. - **MCP mode: only `mcp__heygen__*` tools.** If translation isn't exposed via MCP yet, fall through to CLI for translation operations specifically. Do not synthesize raw HTTP calls. - **CLI mode: only `heygen ...` commands.** Run `heygen video-translate --help` and `heygen video-translate <subcommand> --help` to discover arguments. Use `--request-schema` to see the full JSON shape of any create command. - **Operations below show MCP and CLI side-by-side** — read only the column for your detected mode. ### MCP tool names (MCP mode only) `create_video_translation` (single language). Multi-language and proofreads are not yet exposed via MCP — fall through to CLI for those. Run `mcp__heygen__*` tool listing at session start to confirm what's available; tool surface evolves. ### CLI command groups (CLI mode only) ``` heygen video-translate ├── languages list # supported target languages ├── create # submit a translation job (single or batch) ├── get <id> # check status / fetch result ├── list # list past translations ├── update # update job metadata ├── delete # delete a job └── proofreads ├── create # extract editable subtitles before final render ├── get <id> # check proofread session status ├── srt get <id> # download the extracted SRT ├── srt update <id> # upload edited SRT └── generate <id> # render the final video from approved SRT heygen asset create --file <path> # for local source video uploads (max 32 MB) ``` Every command supports `--help`. Use `--request-schema` on any `create` to see the full JSON body. CLI output: JSON on stdout, `{error:{code,message,hint}}` envelope on stderr, exit codes `0` ok · `1` API · `2` usage · `3` auth · `4` timeout. Add `--wait` on `create` to block until the job completes (default timeout 20m). 📖 **Detailed CLI/MCP error → action mapping → [references/troubleshooting.md](references/troubleshooting.md)** --- ## Default Workflow The skill runs four phases. Phase 1 (Discovery) is the only place you ask questions. Phase 2 (Pre-flight) is silent. Phase 3 (Submit + Poll) is silent. Phase 4 (Deliver) is one short message. ``` Phase 1 — Discovery — gather minimum needed inputs from the user Phase 2 — Pre-flight — validate language, classify content, set flags Phase 3 — Submit + Poll — kick off, background poll, surface only on done/fail Phase 4 — Deliver — post the result with one-line summary ``` ### Phase 1 — Discovery Ask only what you don't already have. Communicate in the user's language. **Never run a form.** One or two questions per turn, max. **Required inputs (block until you have these):** 1. **Source video.** Public URL, local file path, or a HeyGen asset_id from a previous step. If the user hasn't supplied it, ask: *"What's the source video — a URL, file path, or an existing HeyGen asset?"* 2. **Target language(s).** Ask as an open-ended question in the user's language: *"Which language should I translate it into?"* Do NOT present a picker or pre-assigned choices — let the user type freely. They may want one language, multiple, or a region-specific variant. Accept whatever they give and validate against the canonical languages list in Phase 2. **Important inputs (ask if not provided, with smart defaults):** 3. **Speaker count.** Single speaker is the default and what most users have. Ask once when ambiguous: *"How many distinct speakers are in the video?"* Wrong speaker count is the #1 quality killer — speaker confusion creates voice swaps mid-translation. Don't skip this for multi-person content. 4. **Content type.** You usually don't need to ask — infer from the video and confirm. The five profiles below cover ~95% of cases. Only ask if genuinely ambiguous. 5. **Caption preference.** Default ON for talking-head and corporate; default OFF for podcast/audio-only. If you flip the default, mention it briefly in Phase 4. 6. **Duration flexibility.** Ask: *"Does the translated video need to be exactly the same length as the original, or can it be slightly longer/shorter? Allowing flexibility usually sounds more natural — the translated speech gets enough room to be spoken at a comfortable pace instead of being sped up or compressed."* Default recommendation: flexible (`enable_dynamic_duration: true`). Only set to `false` when the user needs frame-exact timing (e.g., syncing to a timeline, ad slot, or external audio track). **Optional (ask only if relevant):** 7. **Glossary / do-not-translate terms.** For corporate or technical content, ask: *"Any product names, company names, or jargon I should keep in the original language?"* HeyGen doesn't currently accept a hard glossary, so this becomes guidance for the proofread step (Phase 3-Proofread) when stakes are high. 8. **Partial translation.** If the user mentions a specific segment ("just the intro", "from 1:30 to 4:00"), capture `start_time` and `end_time` in seconds. 9. **Proofread before final render?** Default OFF (faster, fewer approvals). Default ON for: long videos (>3 min), corporate/branded content, high-stakes legal/medical/educational, languages the user reads natively (so they can verify). Ask: *"Want to review and edit the subtitles before final render? Adds about 5 minutes but lets you fix any wrong terms."* 📖 **Locale-pair gotchas (formality registers, RTL languages, tonal compression, lip-sync ceiling) → [references/language-locale-guide.md](references/language-locale-guide.md)** ### Phase 2 — Pre-flight Silent. No user-facing chatter. Three checks, in order. **Check 2a: Language validation.** **MCP:** `list_video_translation_languages()` (if exposed). Otherwise CLI. **CLI:** `heygen video-translate languages list | jq -r '.data.languages[]'` The list contains exact strings ("Spanish (Spain)", "Chinese (Mandarin, Simplified)", "Arabic (Saudi Arabia)"). Match the user's input case-insensitively against these exact strings. If they say "Spanish", default to "Spanish (Spain)" and confirm in Phase 4. If they say "Chinese", default to "Chinese (Mandarin, Simplified)". If they specify a region ("Mexican Spanish"), map it ("Spanish (Mexico)"). If no match: ask the user to pick from the closest options. **Check 2b: Source video routing.** | Source the user gave you | Route | |--------------------------|-------| | Public HTTPS URL (no auth, returns video MIME on `HEAD`) | Pass directly as `{type: "url", url: "..."}` | | Auth-walled URL, 403, 404, or HTML response | Tell the user, ask for a public URL or local file | | Local file path | Upload via `heygen asset create --file <path>` (CLI) or `upload_asset` (MCP). Max 32 MB. Use the returned `asset_id` as `{type: "asset_id", asset_id: "..."}` | | Existing HeyGen asset_id | Pass directly as `{type: "asset_id", asset_id: "..."}` | 📖 **Asset routing edge cases (very large files, presigned URLs, auth-walled sources) → [references/asset-routing.md](references/asset-routing.md)** **Check 2c: Content profile.** Pick one profile based on the source. Don't list all five to the user — propose silently and only ask if the source is genuinely ambiguous (e.g., a music-heavy talking-head where you can't tell if speech enhancement will help). | Profile | Use when | Flags | |---------|----------|-------| | **Talking head / presenter** (default) | One person speaks to camera; clean audio | `mode: precision`, `enable_speech_enhancement: true`, `enable_caption: true`, `enable_dynamic_duration: true`, `keep_the_same_format: true` | | **Podcast / audio-only** | The visual is static, doesn't matter, or doesn't exist | `mode: precision`, `translate_audio_only: true`, `enable_speech_enhancement: true`, `enable_caption: true` | | **Music / high-soundtrack** | Background music interferes with speech | `mode: precision`, `disable_music_track: true`, `enable_speech_enhancement: true`, `enable_dynamic_duration: true`, `keep_the_same_format: true` | | **Multi-speaker** | Two or more distinct speakers | Talking-head defaults + `speaker_num: <count>`. Speaker count is REQUIRED here — don't guess. | | **Corporate / branded** | Brand voice, glossary discipline, high-stakes | Talking-head defaults + (if user has one) `brand_voice_id`. Strongly consider proofreads for this profile. | **Always:** - `mode: "precision"` unless the user explicitly asks for "fast" / "quick" / "speed". - `enable_dynamic_duration`: set based on the user's answer to the duration flexibility question in Phase 1. Default `true` (recommended) — lets translated speech breathe instead of being crammed into the source's exact timing. Set `false` only when the user explicitly needs fixed-length output. Tonal compression makes flexibility especially important for en→zh, en→ja, en→ko (Asian languages run shorter); de→en, ja→en (run longer); ar/he/ur (RTL + register shifts). - `keep_the_same_format: true` for visual translations — preserves the source's resolution and bitrate so the dubbed video matches the original's encoding. - `enable_watermark: false` (the default). ### Phase 3 — Submit + Poll Silent. Background work. Surface only on (a) per-language completion, (b) per-language hard failure, (c) >5 min progress check. **Branching:** - **Standard path** (proofread = OFF): submit translations, background-poll, deliver. - **Proofread path** (proofread = ON): create proofread session → download SRT → user edits or you assist → upload edited SRT → generate final → background-poll → deliver. #### Standard path Submit one job per target language using batch syntax (`--output-languages` accepts multiple). **MCP** *(single language only at time of writing)*: ``` create_video_translation( video={type, url|asset_id}, output_languages=["Spanish (Spain)"], mode="precision", enable_speech_enhancement=true, enable_caption=true, enable_dynamic_duration=true, keep_the_same_format=true, speaker_num=<n>, # only when known multi-speaker ) ``` **CLI:** ```bash heygen video-translate create \ -d '{"video":{"type":"url","url":"https://..."},"output_languages":["Spanish (Spain)","Japanese (Japan)"]}' \ --mode precision \ --enable-speech-enhancement \ --enable-caption \ --enable-dynamic-duration \ --keep-the-same-format \ --speaker-num 1 \ --title "<short title>" ``` Response returns one `video_translation_id` per language. Capture all of them. **Polling (silent, backgrounded):** Use `--wait` on `create` to block until completion when running ONE language. For batch, drop `--wait` and poll each ID: ```bash # CLI mode polling (background) heygen video-translate get <video-translation-id> # Returns { data: { status: "pending"|"running"|"completed"|"failed", video_url, ... } } ``` Polling cadence: 30s for the first 3 minutes, then 60s. Most translations complete in 5–15 min; some (long videos, batched languages) take 30+ min. Hard timeout: 60 min per translation — beyond that, treat as stuck and surface the issue. **MCP equivalents:** `get_video_translation(id)` (if exposed). Otherwise fall through to CLI for polling. 📖 **Background polling pattern (don't poll in foreground / harness-specific notes) → [references/troubleshooting.md#polling](references/troubleshooting.md)** #### Proofread path For high-stakes content, run a proofread session first so the user can review/edit the translated subtitles before the engine commits to a final render. ```bash # 1. Create proofread session — returns proofread_ids (one per language) heygen video-translate proofreads create \ -d '{"video":{"type":"url","url":"https://..."}}' \ --output-languages "Spanish (Spain)" \ --mode precision \ --enable-speech-enhancement \ --keep-the-same-format \ --speaker-num 1 \ --title "<short fileNAME-safe title>" # → status: processing (3–5 min for short videos)