minimax-multimodal-toolkit

# MiniMax Multi-Modal Toolkit Generate voice, music, video, and image content via MiniMax APIs — the unified entry for **MiniMax multimodal** use cases (audio + music + video + image). Includes voice cloning & voice design for custom voices, image generation with character reference, and FFmpeg-based media tools for audio/video format conversion, concatenation, trimming, and extraction. ## Output Directory **All generated files MUST be saved to `minimax-output/` under the AGENT'S current working directory (NOT the skill directory).** Every script call MUST include an explicit `--output` / `-o` argument pointing to this location. Never omit the output argument or rely on script defaults. **Rules:** 1. Before running any script, ensure `minimax-output/` exists in the agent's working directory (create if needed: `mkdir -p minimax-output`) 2. Always use absolute or relative paths from the agent's working directory: `--output minimax-output/video.mp4` 3. **Never** `cd` into the skill directory to run scripts — run from the agent's working directory using the full script path 4. Intermediate/temp files (segment audio, video segments, extracted frames) are automatically placed in `minimax-output/tmp/`. They can be cleaned up when no longer needed: `rm -rf minimax-output/tmp` ## Prerequisites ```bash brew install ffmpeg jq # macOS (or apt install ffmpeg jq on Linux) bash scripts/check_environment.sh ``` No Python or pip required — all scripts are pure bash using `curl`, `ffmpeg`, `jq`, and `xxd`. ### API Host Configuration MiniMax provides two service endpoints for different regions. Set `MINIMAX_API_HOST` before running any script: | Region | Platform URL | API Host Value | |--------|-------------|----------------| | China Mainland（中国大陆） | https://platform.minimaxi.com | `https://api.minimaxi.com` | | Global（全球） | https://platform.minimax.io | `https://api.minimax.io` | ```bash # China Mainland export MINIMAX_API_HOST="https://api.minimaxi.com" # or Global export MINIMAX_API_HOST="https://api.minimax.io" ``` **IMPORTANT — When API Host is missing:** Before running any script, check if `MINIMAX_API_HOST` is set in the environment. If it is NOT configured: 1. Ask the user which service endpoint their MiniMax account uses: - **China Mainland** → `https://api.minimaxi.com` - **Global** → `https://api.minimax.io` 2. Instruct and help user to set it via `export MINIMAX_API_HOST="https://api.minimaxi.com"` (or the global variant) in their terminal or add it to their shell profile (`~/.zshrc` / `~/.bashrc`) for persistence ### API Key Configuration Set the `MINIMAX_API_KEY` environment variable before running any script: ```bash export MINIMAX_API_KEY="your-api-key-here" ``` The key starts with `sk-api-` or `sk-cp-`, obtainable from https://platform.minimaxi.com (China) or https://platform.minimax.io (Global) **IMPORTANT — When API Key is missing:** Before running any script, check if `MINIMAX_API_KEY` is set in the environment. If it is NOT configured: 1. Ask the user to provide their MiniMax API key 2. Instruct and help user to set it via `export MINIMAX_API_KEY="sk-..."` in their terminal or add it to their shell profile (`~/.zshrc` / `~/.bashrc`) for persistence ## Plan Limits & Quotas **IMPORTANT — Always respect the user's plan limits before generating content.** If the user's quota is exhausted or insufficient, warn them before proceeding. ### Standard Plans | Capability | Starter | Plus | Max | |---|---|---|---| | M2.7 (chat) | 600 req/5h | 1,500 req/5h | 4,500 req/5h | | Speech 2.8 | — | 4,000 chars/day | 11,000 chars/day | | image-01 | — | 50 images/day | 120 images/day | | Hailuo-2.3-Fast 768P 6s | — | — | 2 videos/day | | Hailuo-2.3 768P 6s | — | — | 2 videos/day | | Music-2.5 | — | — | 4 songs/day (≤5 min each) | ### High-Speed Plans | Capability | Plus-HS | Max-HS | Ultra-HS | |---|---|---|---| | M2.7-highspeed (chat) | 1,500 req/5h | 4,500 req/5h | 30,000 req/5h | | Speech 2.8 | 9,000 chars/day | 19,000 chars/day | 50,000 chars/day | | image-01 | 100 images/day | 200 images/day | 800 images/day | | Hailuo-2.3-Fast 768P 6s | — | 3 videos/day | 5 videos/day | | Hailuo-2.3 768P 6s | — | 3 videos/day | 5 videos/day | | Music-2.5 | — | 7 songs/day (≤5 min each) | 15 songs/day (≤5 min each) | **Key quota constraints:** - **Video resolution: 768P only** — 1080P is not available on any plan - **Video duration: 6s** — all plan quotas are counted in 6-second units - **Video quota is very limited** (2–5/day depending on plan) — always confirm with the user before generating video ## Key Capabilities | Capability | Description | Entry point | |------------|-------------|-------------| | TTS | Text-to-speech synthesis with multiple voices and emotions | `scripts/tts/generate_voice.sh` | | Voice Cloning | Clone a voice from an audio sample (10s–5min) | `scripts/tts/generate_voice.sh clone` | | Voice Design | Create a custom voice from a text description | `scripts/tts/generate_voice.sh design` | | Music Generation | Generate songs with lyrics or instrumental tracks | `scripts/music/generate_music.sh` | | Image Generation | Text-to-image, image-to-image with character reference | `scripts/image/generate_image.sh` | | Video Generation | Text-to-video, image-to-video, subject reference, templates | `scripts/video/generate_video.sh` | | Long Video | Multi-scene chained video with crossfade transitions | `scripts/video/generate_long_video.sh` | | Media Tools | Audio/video format conversion, concatenation, trimming, extraction | `scripts/media_tools.sh` | ## TTS (Text-to-Speech) Entry point: `scripts/tts/generate_voice.sh` ### IMPORTANT: Single voice vs Multi-segment — Choose the right approach | User intent | Approach | |-------------|----------| | Single voice / no multi-character need | `tts` command — generate the entire text in one call | | Multiple characters / narrator + dialogue | `generate` command with segments.json | **Default behavior:** When the user simply asks to generate speech/voice and does NOT mention multiple voices or characters, use the `tts` command directly with a single appropriate voice. Do NOT split into segments or use the multi-segment pipeline — just pass the full text to `tts` in one call. Only use multi-segment `generate` when: - The user explicitly needs multiple voices/characters - The text requires narrator + character dialogue separation - The text exceeds **10,000 characters** (API limit per request) — in this case, split into segments with the same voice ### Single-voice generation (DEFAULT) ```bash bash scripts/tts/generate_voice.sh tts "Hello world" -o minimax-output/hello.mp3 bash scripts/tts/generate_voice.sh tts "你好世界" -v female-shaonv -o minimax-output/hello_cn.mp3 ``` ### Multi-segment generation (multi-voice / audiobook / podcast) **Complete workflow — follow ALL steps in order:** 1. **Write segments.json** — split text into segments with voice assignments (see format and rules below) 2. **Run `generate` command** — this reads segments.json, generates audio for EACH segment via TTS API, then merges them into a single output file with crossfade ```bash # Step 1: Write segments.json to minimax-output/ # (use the Write tool to create minimax-output/segments.json) # Step 2: Generate audio from segments.json — this is the CRITICAL step # It generates each segment individually and merges them into one file bash scripts/tts/generate_voice.sh generate minimax-output/segments.json \ -o minimax-output/output.mp3 --crossfade 200 ``` **Do NOT skip Step 2.** Writing segments.json alone does nothing — you MUST run the `generate` command to actually produce audio. ### Voice management ```bash # List all available voices bash scripts/tts/generate_voice.sh list-voices # Voice cloning (from audio sample, 10s–5min) bash scripts/tts/generate_voice.sh clone sample.mp3 --voice-id my-voice # Voice design (from text description) bash scripts/tts/generate_voice.sh design "A warm female narrator voice" --voice-id narrator ``` ### Audio processing ```bash bash scripts/tts/generate_voice.sh merge part1.mp3 part2.mp3 -o minimax-output/combined.mp3 bash scripts/tts/generate_voice.sh convert input.wav -o minimax-output/output.mp3 ``` ### TTS Models | Model | Notes | |-------|-------| | speech-2.8-hd | Recommended, auto emotion matching | | speech-2.8-turbo | Faster variant | | speech-2.6-hd | Previous gen, manual emotion | | speech-2.6-turbo | Previous gen, faster | ### segments.json Format Default crossfade between segments: **200ms** (`--crossfade 200`). ```json [ { "text": "Hello!", "voice_id": "female-shaonv", "emotion": "" }, { "text": "Welcome.", "voice_id": "male-qn-qingse", "emotion": "happy" } ] ``` Leave `emotion` empty for speech-2.8 models (auto-matched from text). ### IMPORTANT: Multi-Segment Script Generation Rules (Audiobooks, Podcasts, etc.) When generating segments.json for audiobooks, podcasts, or any multi-character narration, you MUST split narration text from character dialogue into separate segments with distinct voices. **Rule: Narration and dialogue are ALWAYS separate segments.** A sentence like `"Tom said: The weather is great today!"` must be split into two segments: - Segment 1 (narrator voice): `"Tom said:"` - Segment 2 (character voice): `"The weather is great today!"` **Example — Audiobook with narrator + 2 characters:** ```json [ { "text": "Morning sunlight streamed into the classroom as students filed in one by one.", "voice_id": "narrator-voice", "emotion": "" }, { "text": "Tom smiled and turned to Lisa:", "voice_id": "narrator-voice", "emotion": "" }, { "text": "The weather is amazing today! Let's go to the park after school!", "voice_id": "tom-voice", "emotion": "happy" }, { "text": "Lisa thought for a moment, then replied:", "voice_id": "narrator-voice", "emotion": "" }, { "text": "Sure, but I need to drop off my backpack at home first.", "voice_id": "lisa-voice", "emotion": "" }, { "text": "They exchanged a smile and went back to listening to the lecture.", "voice_id": "narrator-voice", "emotion": "" } ] ``` **Key principles:** 1. **Narrator** uses a consistent neutral narrator voice throughout 2. **Each character** has a dedicated voice_id, maintained consistently across all their dialogue 3. **Split at dialogue boundaries** — `"He said:"` is narrator, the quoted content is the character 4. **Do NOT merge** narrator text and character speech into a single segment 5. For characters without pre-existing voice_ids, use voice cloning or voice design to create them first, then reference the created voice_id in segments ## Music Generation Entry point: `scripts/music/generate_music.sh` ### IMPORTANT: Instrumental vs Lyrics — When to use which | Scenario | Mode | Action | |----------|------|--------| | BGM for video / voice / podcast | Instrumental (default) | Use `--instrumental` directly, do NOT ask user | | User explicitly asks to "create music" / "make a song" | Ask user first | Ask whether they want instrumental or with lyrics | **When adding background music to video or voice content**, always default to instrumental mode (`--instrumental`). Do not ask the user — BGM should never have vocals competing with the main content. **When the user explicitly asks to create/generate music as the primary task**, ask them whether they want: - Instrumental (pure music, no vocals) - With lyrics (song with vocals — user provides or you help write lyrics) ```bash # Instrumental (for BGM or when user chooses instrumental) bash scripts/music/generate_music.sh \ --instrumental \ --prompt "ambient electronic, atmospheric" \ --output minimax-output/ambient.mp3 --download # Song with lyrics (when user chooses vocal music) bash scripts/music/generate_music.sh \ --lyrics "[verse]\nHello world\n[chorus]\nLa la la" \ --prompt "indie folk, melancholic" \ --output minimax-output/song.mp3 --download # With style fields bash scripts/music/generate_music.sh \ --lyrics "[verse]\nLyrics here" \ --genre "pop" --mood "upbeat" --tempo "fast" \ --output minimax-output/pop_track.mp3 --download ``` ### Music Model Default model: `music-2.5` `music-2.5` does **not** support `--instrumental` directly. When instrumental music is needed, the script automatically applies a workaround: - Sets lyrics to `[intro] [outro]` (empty structural tags, no actual vocals), appends `pure music, no lyrics` to the prompt This produces instrumental-style output without requiring manual intervention. You can always use `--instrumental` and the script handles the rest. ## Image Generation Entry point: `scripts/image/generate_image.sh` Model: `image-01` — photorealistic image generation from text prompts, with optional character reference for image-to-image. ### IMPORTANT: Mode Selection — t2i vs i2i | User intent | Mode | |-------------|------| | Generate image from text description (default) | `t2i` — text-to-image | | Generate image with a character reference photo (keep same person) | `i2i` — image-to-image | **Default behavior:** When the user asks to generate/create an image without mentioning a reference photo, use `t2i` mode (default). Only use `i2i` mode when the user provides a character reference image or explicitly asks to base the image on an existing person's appearance.