| name | mk-multimodal |
| description | Process images/video/audio/PDFs via Gemini API; generate images/video/speech/music; convert docs to Markdown. Multi-provider fallback. Use for analyze/transcribe/generate/convert on media files. |
Multimodal Analysis & Generation
Path convention: Commands below assume cwd is $(git rev-parse --show-toplevel) (project root). Prefix paths with "$(git rev-parse --show-toplevel)/" when invoking from subdirectories.
Model selection and commands
Use the configured analysis default for media understanding. Generation routes Gemini → MiniMax →
OpenRouter as keys permit. Provider model IDs, preview availability, and pricing change frequently:
load references/models-and-pricing.md at execution time and
verify preview IDs against the provider dashboard before relying on one.
| Task | Script |
|---|
| Analyze, transcribe, extract | scripts/gemini_analyze.py |
| Generate image or video with routing | scripts/gemini_generate.py |
| Generate MiniMax image, video, speech, music | scripts/minimax_generate.py |
| Convert document to Markdown | scripts/document_converter.py |
| Validate keys and dependencies | scripts/check_setup.py |
.agents/skills/.venv/bin/python3 .agents/skills/multimodal/scripts/gemini_analyze.py \
--files <path> --task <analyze|transcribe|extract> [--resolution low-res] [--json]
References
Load only when executing the corresponding step.
| Reference | When to load | Content |
|---|
| vision-understanding.md | Image analysis | Detection, segmentation, OCR |
| audio-processing.md | Audio/video transcription | Formats, splitting, timestamps |
| models-and-pricing.md | Model selection, cost | Full model table, pricing |
| image-generation.md | Image generation | Nano Banana 2, OpenRouter fallback |
| video-generation.md | Video generation | Veo 3.1, async polling |
| video-analysis.md | Video analysis | Resolution modes, cost math |
| minimax-generation.md | MiniMax generation | Image, video, TTS, music |
| document-conversion.md | Document conversion | Formats, batch mode |
When to Use
Auto-activate on these patterns:
- Task references image (.png, .jpg, .webp), video (.mp4, .mov), audio (.mp3, .wav), or document (.pdf, .docx) files
- Task asks to "analyze", "describe", "transcribe", "extract", "OCR"
- Task asks to "generate image", "generate video", "create image"
- Task asks to "generate speech", "text to speech", "TTS"
- Task asks to "generate music", "create music"
- Task asks to "convert document", "convert PDF to markdown"
- User references a non-text binary file for processing
Do NOT invoke when: text-only files (use Read), Gemini API docs (use mk:docs-finder), already-described image in context.
API key check
Use MEOWKIT_GEMINI_API_KEY (or legacy GEMINI_API_KEY); without it, generation may use
MEOWKIT_MINIMAX_API_KEY. If neither is present, stop with setup instructions.
Analysis Process
Detect media format → select model (default: gemini-2.5-flash) → run script. For >20MB files, use File API. Use --resolution low-res for video (62% savings, video only). Output capped at ~3000 tokens; prefer structured JSON/Markdown.
Generation Process
Provider router auto-selects best available provider by checking API keys:
- Image: Gemini → MiniMax → OpenRouter
- Video: Gemini → MiniMax
- TTS: MiniMax only
- Music: MiniMax only
Force a provider: --provider gemini|minimax|openrouter. Override chain order via MEOWKIT_IMAGE_PROVIDER_CHAIN etc.
Gotchas
- Python venv required: if you get
python3: command not found or import errors, run .codex/scripts/bin/setup-workflow once from the project root.
- Audio >15 min: Gemini truncates silently. Split first.
- PDF >100 pages: Quality degrades. Process in 20-page chunks.
- Video cost: ~263 tokens/sec at default resolution. Use
--verbose for cost estimate.
- Image gen requires billing: Free tier = no gen. Use MiniMax/OpenRouter as fallback.
- MiniMax video timeout: Hailuo takes 60-180s. Max 600s.
- TTS voices: load the provider catalog at execution time; voice inventories change.
- Temperature: Keep Gemini at 1.0. Lowering causes degraded output.
Failure Handling
- Missing API key → STOP + setup instructions
- Invalid key (401) → STOP + re-check key
- Rate limit (429) → key rotation (
MEOWKIT_GEMINI_API_KEY_2/3/4), else wait 60s
- Billing required → provider router tries MiniMax/OpenRouter fallback
Setup
Set MEOWKIT_GEMINI_API_KEY in env or .env file. Legacy GEMINI_API_KEY also works.
Optional: MEOWKIT_MINIMAX_API_KEY for TTS/music and alternative generation.
Budget rule: ≤3000 tokens inline. If response exceeds budget, summarize key findings before returning.
