openclaw-videotranslate-skill

# OpenClaw Video Translation & Dubbing Skill > Skill by [ara.so](https://ara.so) — Hermes Skills collection. A professional-grade OpenClaw skill for translating video subtitles and generating dubbed audio across multiple languages. Features a 3D adaptive scheduler that handles API rate limits, automatic audio duration alignment, and lossless multi-track video output. ## What It Does This skill processes videos in two modes: 1. **Subtitle-only mode**: Translates subtitles while keeping original audio, outputs dual-subtitle video 2. **Subtitle + Dubbing mode**: Translates subtitles AND generates TTS audio in the target language, with automatic timing alignment Key capabilities: - Extracts embedded or external subtitles (SRT/VTT) - Translates using LLM or web translation APIs - Generates natural-sounding TTS audio with voice mapping - Automatically stretches/compresses audio to match original timing - Outputs lossless multi-track MKV videos - Handles API rate limits with adaptive 3D scheduling (batch size, payload size, concurrency) ## Installation ### Prerequisites ```bash # Install FFmpeg (required for video processing) # macOS: brew install ffmpeg # Ubuntu/Debian: sudo apt install ffmpeg # Windows (use Chocolatey): choco install ffmpeg ``` ### Install the Skill ```bash # Clone the repository git clone https://github.com/zbjincheng/openclaw-skill-videotranslate.git cd openclaw-skill-videotranslate # Install with pip pip install -e . # Or with development dependencies pip install -e ".[dev]" ``` ### Environment Setup ```bash # Copy environment template cp .env.example .env ``` Edit `.env` to add your API credentials: ```bash # Translation API (LLM or Web service) TRANSLATION_ENDPOINT=https://api.openai.com/v1/chat/completions TRANSLATION_CREDENTIAL=your_api_key_here # TTS API (for dubbing mode) TTS_ENDPOINT=https://api.openai.com/v1/audio/speech TTS_CREDENTIAL=your_api_key_here ``` ## Configuration The skill is configured via `manifest.yaml`. Here's a complete example: ```yaml # manifest.yaml skill: name: translation_dubbing_skill version: "1.0.0" parameters: # Required: Input video file video_path: type: path required: true description: "Path to source video file" # Optional: External subtitle file (extracts embedded if omitted) subtitle_path: type: path required: false description: "Path to .srt or .vtt subtitle file" # Required: Source and target languages source_language: type: string required: true default: "en" description: "Source language code (en, zh-CN, ja, es, fr, de, ko)" target_language: type: string required: true default: "zh-CN" description: "Target language code for translation" # Required: Processing mode processing_mode: type: enum required: true default: "subtitle_and_dubbing" values: - subtitle_only - subtitle_and_dubbing # Required: Translation provider translation_provider: type: enum required: true values: [llm, web] translation_endpoint: type: string required: true description: "Translation API endpoint" translation_credential: type: secret required: true description: "API key for translation service" # Required if processing_mode is subtitle_and_dubbing tts_provider: type: enum required: false values: [llm, web] tts_endpoint: type: string required: false description: "TTS API endpoint" tts_credential: type: secret required: false description: "API key for TTS service" # Optional: Advanced scheduler tuning scheduler: max_concurrency: 10 initial_batch_size: 20 max_payload_tokens: 4000 backoff_multiplier: 2.0 ``` ## Usage Patterns ### Basic Subtitle Translation (No Dubbing) ```python from translation_dubbing_skill.entry import TranslationDubbingSkill from translation_dubbing_skill.models import ProcessingMode # Initialize skill skill = TranslationDubbingSkill( video_path="input.mp4", source_language="en", target_language="zh-CN", processing_mode=ProcessingMode.SUBTITLE_ONLY, translation_provider="llm", translation_endpoint="https://api.openai.com/v1/chat/completions", translation_credential="${TRANSLATION_CREDENTIAL}", output_dir="./output" ) # Run translation result = skill.execute() print(f"Output video: {result['output_video']}") print(f"Subtitle file: {result['output_subtitle']}") ``` ### Full Translation + Dubbing ```python from translation_dubbing_skill.entry import TranslationDubbingSkill from translation_dubbing_skill.models import ProcessingMode skill = TranslationDubbingSkill( video_path="lecture.mp4", subtitle_path="lecture.srt", # Optional: use external subtitles source_language="en", target_language="ja", processing_mode=ProcessingMode.SUBTITLE_AND_DUBBING, # Translation config translation_provider="llm", translation_endpoint="https://api.openai.com/v1/chat/completions", translation_credential="${TRANSLATION_CREDENTIAL}", # TTS config tts_provider="llm", tts_endpoint="https://api.openai.com/v1/audio/speech", tts_credential="${TTS_CREDENTIAL}", output_dir="./output" ) # Execute with progress tracking result = skill.execute() ``` ### Using External Subtitle Files ```python # When you already have subtitles extracted skill = TranslationDubbingSkill( video_path="video.mp4", subtitle_path="video.srt", # Provide external .srt or .vtt source_language="en", target_language="es", processing_mode=ProcessingMode.SUBTITLE_ONLY, translation_provider="web", translation_endpoint="https://translation-api.example.com/translate", translation_credential="${WEB_TRANSLATION_KEY}" ) ``` ### Custom Scheduler Configuration ```python # Fine-tune for rate-limited APIs skill = TranslationDubbingSkill( video_path="long_video.mp4", source_language="en", target_language="zh-CN", processing_mode=ProcessingMode.SUBTITLE_AND_DUBBING, translation_provider="llm", translation_endpoint="${TRANSLATION_ENDPOINT}", translation_credential="${TRANSLATION_CREDENTIAL}", tts_provider="llm", tts_endpoint="${TTS_ENDPOINT}", tts_credential="${TTS_CREDENTIAL}", # Scheduler tuning scheduler_config={ "max_concurrency": 5, # Lower for strict rate limits "initial_batch_size": 10, # Start with smaller batches "max_payload_tokens": 2000, # Reduce token count per request "backoff_multiplier": 3.0, # More aggressive backoff on 429 "max_retries": 5 } ) ``` ## Working with Subtitles ### Parsing Subtitle Files ```python from translation_dubbing_skill.subtitle import SubtitleParser # Parse SRT or VTT parser = SubtitleParser() entries = parser.parse("video.srt") for entry in entries: print(f"[{entry.start_time} -> {entry.end_time}]") print(f" {entry.text}") print(f" Speaker: {entry.speaker_id}") ``` ### Creating Subtitle Objects ```python from translation_dubbing_skill.models import SubtitleEntry entry = SubtitleEntry( index=1, start_time="00:00:01,500", end_time="00:00:04,200", text="Hello, welcome to the tutorial.", speaker_id="narrator", duration_ms=2700 ) ``` ### Writing Translated Subtitles ```python from translation_dubbing_skill.subtitle import SubtitleSerializer # After translation translated_entries = [ SubtitleEntry( index=1, start_time="00:00:01,500", end_time="00:00:04,200", text="你好,欢迎来到教程。", speaker_id="narrator", duration_ms=2700 ) ] serializer = SubtitleSerializer() serializer.write("output_zh.srt", translated_entries, format="srt") ``` ## Translation Providers ### LLM Provider (OpenAI-compatible) ```python from translation_dubbing_skill.providers.translation import LLMTranslationProvider provider = LLMTranslationProvider( endpoint="https://api.openai.com/v1/chat/completions", credential="${OPENAI_API_KEY}", model="gpt-4o", source_language="en", target_language="zh-CN" ) # Translate batch of entries translated = await provider.translate_batch(subtitle_entries) ``` The LLM provider: - Automatically strips `<think>` reasoning blocks - Parses JSON responses robustly - Handles context window overflow with payload slicing - Supports custom system prompts for translation style ### Web Provider (Generic Translation API) ```python from translation_dubbing_skill.providers.translation import WebTranslationProvider provider = WebTranslationProvider( endpoint="https://api.translator.com/v1/translate", credential="${WEB_API_KEY}", source_language="en", target_language="fr" ) translated = await provider.translate_batch(subtitle_entries) ``` ## TTS Providers ### LLM TTS Provider (OpenAI-compatible) ```python from translation_dubbing_skill.providers.tts import LLMTTSProvider provider = LLMTTSProvider( endpoint="https://api.openai.com/v1/audio/speech", credential="${OPENAI_API_KEY}", model="tts-1-hd", voice_mapping={ "narrator": "alloy", "speaker1": "echo", "speaker2": "nova" } ) # Generate audio for translated subtitle audio_bytes = await provider.synthesize( text="你好,欢迎来到教程。", speaker_id="narrator", target_duration_ms=2700 # Auto-stretches to match original ) ``` ### Web TTS Provider ```python from translation_dubbing_skill.providers.tts import WebTTSProvider provider = WebTTSProvider( endpoint="https://tts-api.example.com/synthesize", credential="${TTS_API_KEY}", voice_mapping={"default": "zh-CN-XiaoxiaoNeural"} ) audio = await provider.synthesize( text="翻译后的文本", speaker_id="default", target_duration_ms=3000 ) ``` ## Audio Duration Alignment The skill automatically aligns TTS audio to match original subtitle timing: ```python from translation_dubbing_skill.align import AudioAligner aligner = AudioAligner() # Stretch or compress audio to exact duration aligned_audio = aligner.align_duration( audio_bytes=tts_output, target_duration_ms=2700, original_duration_ms=3200 # TTS was 3.2s, needs to be 2.7s ) ``` Uses time-stretching (not pitch-shifted) to preserve voice quality. ## Video Muxing ### Creating Multi-Track Output ```python from translation_dubbing_skill.mux import VideoMuxer muxer = VideoMuxer() output_path = muxer.mux_video( video_path="original.mp4", original_audio_path="original_audio.aac", dubbed_audio_path="dubbed_zh.mp3", original_subtitle_path="original_en.srt", translated_subtitle_path="translated_zh.srt", output_path="output_dual_track.mkv", source_language="en", target_language="zh-CN" ) ``` Output structure: - **Video**: Original video stream (lossless copy) - **Audio Track 1** (default): Dubbed target language - **Audio Track 2**: Original source language - **Subtitle Track 1** (default): Translated target language - **Subtitle Track 2**: Original source language ## Progress Tracking ### Monitor Processing Status ```python from translation_dubbing_skill.progress import ProgressListener class MyProgressListener(ProgressListener): def on_state_change(self, stage: str, current: int, total: int, message: str): print(f"[{stage}] {current}/{total} - {message}") def on_error(self, error: Exception): print(f"Error: {error}") def on_complete(self, result: dict): print(f"Complete: {result}") skill = TranslationDubbingSkill( video_path="video.mp4", # ... other params progress_listener=MyProgressListener() ) skill.execute() ``` ## Error Handling ### Common Error Types ```python from translation_dubbing_skill.errors import ( SubtitleExtractionError, TranslationAPIError, TTSAPIError, AudioAlignmentError, VideoMuxingError, RateLimitError ) try: result = skill.execute() except SubtitleExtractionError as e: print(f"Could not extract subtitles: {e}") except TranslationAPIError as e: print(f"Translation failed: {e}") print(f"Failed entries: {e.failed_entries}") except TTSAPIError as e: print(f"TTS synthesis failed: {e}") except RateLimitError as e: print(f"Hit API rate limit: {e}")