mk-multimodal

name	mk:multimodal
description	Process images, video, audio, PDFs with Gemini API. Generate images (Nano Banana 2), videos (Veo 3), speech (MiniMax TTS), music (MiniMax). Convert documents to Markdown. Multi-provider fallback (Gemini → MiniMax → OpenRouter). Activate when task references media files, asks to analyze/describe/transcribe/extract/generate/convert, or involves non-text binary files.
version	2.1.0
argument-hint	[file-path] [task]
phase	on-demand
source	local
trust_level	kit-authored
injection_risk	low
requires_env	[{"name":"MEOWKIT_GEMINI_API_KEY","required":true,"description":"Google Gemini API key (falls back to GEMINI_API_KEY)","setup":"Get from https://aistudio.google.com/apikey","fallback":"Analysis requires this key. Generation falls back to MiniMax/OpenRouter if available."},{"name":"MEOWKIT_MINIMAX_API_KEY","required":false,"description":"MiniMax API key for image/video/TTS/music generation","setup":"Get from https://platform.minimax.io/"}]
keywords	["multimodal","gemini","image-gen","video-gen","tts","transcription","document-conversion","vision","image","video","audio","minimax"]
when_to_use	Use when processing images, video, audio, PDFs or generating media via Gemini/MiniMax/OpenRouter. Activate when task references media files.
user-invocable	true
allowed-tools	["Bash","Read","Write","Edit"]
owner	utility
criticality	medium
status	active
runtime	claude-code
requires_external_service	["gemini","minimax"]
default_enabled	false

Multimodal Analysis & Generation

Path convention: Commands below assume cwd is $CLAUDE_PROJECT_DIR (project root). Prefix paths with "$CLAUDE_PROJECT_DIR/" when invoking from subdirectories.

Model selection and commands

Use the configured analysis default for media understanding. Generation routes Gemini → MiniMax → OpenRouter as keys permit. Provider model IDs, preview availability, and pricing change frequently: load references/models-and-pricing.md at execution time and verify preview IDs against the provider dashboard before relying on one.

Task	Script
Analyze, transcribe, extract	scripts/gemini_analyze.py
Generate image or video with routing	scripts/gemini_generate.py
Generate MiniMax image, video, speech, music	scripts/minimax_generate.py
Convert document to Markdown	scripts/document_converter.py
Validate keys and dependencies	scripts/check_setup.py

.claude/skills/.venv/bin/python3 .claude/skills/multimodal/scripts/gemini_analyze.py \
  --files <path> --task <analyze|transcribe|extract> [--resolution low-res] [--json]

References

Load only when executing the corresponding step.

Reference	When to load	Content
vision-understanding.md	Image analysis	Detection, segmentation, OCR
audio-processing.md	Audio/video transcription	Formats, splitting, timestamps
models-and-pricing.md	Model selection, cost	Full model table, pricing
image-generation.md	Image generation	Nano Banana 2, OpenRouter fallback
video-generation.md	Video generation	Veo 3.1, async polling
video-analysis.md	Video analysis	Resolution modes, cost math
minimax-generation.md	MiniMax generation	Image, video, TTS, music
document-conversion.md	Document conversion	Formats, batch mode

When to Use

Auto-activate on these patterns:

• Task references image (.png, .jpg, .webp), video (.mp4, .mov), audio (.mp3, .wav), or document (.pdf, .docx) files
• Task asks to "analyze", "describe", "transcribe", "extract", "OCR"
• Task asks to "generate image", "generate video", "create image"
• Task asks to "generate speech", "text to speech", "TTS"
• Task asks to "generate music", "create music"
• Task asks to "convert document", "convert PDF to markdown"
• User references a non-text binary file for processing

Do NOT invoke when: text-only files (use Read), Gemini API docs (use mk:docs-finder), already-described image in context.

API key check

Use MEOWKIT_GEMINI_API_KEY (or legacy GEMINI_API_KEY); without it, generation may use MEOWKIT_MINIMAX_API_KEY. If neither is present, stop with setup instructions.

Analysis Process

Detect media format → select model (default: gemini-2.5-flash) → run script. For >20MB files, use File API. Use --resolution low-res for video (62% savings, video only). Output capped at ~3000 tokens; prefer structured JSON/Markdown.

Generation Process

Provider router auto-selects best available provider by checking API keys:

• Image: Gemini → MiniMax → OpenRouter
• Video: Gemini → MiniMax
• TTS: MiniMax only
• Music: MiniMax only

Force a provider: --provider gemini|minimax|openrouter. Override chain order via MEOWKIT_IMAGE_PROVIDER_CHAIN etc.

Gotchas

• Python venv required: if you get python3: command not found or import errors, run .claude/scripts/bin/setup-workflow once from the project root.
• Audio >15 min: Gemini truncates silently. Split first.
• PDF >100 pages: Quality degrades. Process in 20-page chunks.
• Video cost: ~263 tokens/sec at default resolution. Use --verbose for cost estimate.
• Image gen requires billing: Free tier = no gen. Use MiniMax/OpenRouter as fallback.
• MiniMax video timeout: Hailuo takes 60-180s. Max 600s.
• TTS voices: load the provider catalog at execution time; voice inventories change.
• Temperature: Keep Gemini at 1.0. Lowering causes degraded output.

Failure Handling

• Missing API key → STOP + setup instructions
• Invalid key (401) → STOP + re-check key
• Rate limit (429) → key rotation (MEOWKIT_GEMINI_API_KEY_2/3/4), else wait 60s
• Billing required → provider router tries MiniMax/OpenRouter fallback

Setup

Set MEOWKIT_GEMINI_API_KEY in env or .env file. Legacy GEMINI_API_KEY also works. Optional: MEOWKIT_MINIMAX_API_KEY for TTS/music and alternative generation.

Budget rule: ≤3000 tokens inline. If response exceeds budget, summarize key findings before returning.