ai-multimodal

name	ai-multimodal
version	2.0.0
description	[AI & Tools] Process and generate multimedia content using Google Gemini API -- vision analysis, audio transcription, video processing, document extraction, image/video generation. Triggers on multimodal, vision API, image recognition, audio transcription, video analysis, gemini, imagen, document extraction.
allowed-tools	Bash, Read, Write, Edit

[IMPORTANT] Use TaskCreate to break ALL work into small tasks BEFORE starting — including tasks for each file read. This prevents context loss from long files. For simple tasks, AI MUST ask user whether to skip.

Quick Summary

Goal: Process and generate multimedia content (images, audio, video, documents) using Google Gemini API via Python scripts.

Workflow:

1. Identify Modality — Match input type to task (analyze, transcribe, extract, generate)
2. Check Limits — Inline max 20MB, File API max 2GB; split large audio at 15min chunks
3. Execute — Run gemini_batch_process.py with appropriate task and files
4. Post-Process — Format output as markdown with timestamps, save generated content

Key Rules:

• Requires GEMINI_API_KEY environment variable
• Always request specific nodes/files, avoid full-file downloads
• Use media_optimizer.py to compress/split files exceeding limits

Be skeptical. Apply critical thinking, sequential thinking. Every claim needs traced proof, confidence percentages (Idea should be more than 80%).

AI Multimodal

Purpose

Process audio, images, videos, and documents or generate images/videos using Google Gemini's multimodal API via bundled Python scripts.

When to Use

• Analyzing images or screenshots (Gemini vision is preferred over Claude's built-in vision for complex tasks)
• Transcribing audio files (meetings, podcasts, interviews)
• Extracting data from PDFs, scanned documents, or charts
• Processing video content (scene detection, temporal Q&A)
• Generating images with Imagen 4 or videos with Veo 3
• Converting documents to markdown with visual understanding

When NOT to Use

• Simple text-only LLM calls -- use Claude directly
• Reading a file Claude can already read (code, markdown, JSON) -- use Read tool
• Building AI-powered application features -- use api-design or frontend-design
• Music composition workflows -- load references/music-generation.md only when specifically requested
• General prompt engineering -- use ai-artist skill

Prerequisites

export GEMINI_API_KEY="your-key"  # From https://aistudio.google.com/apikey
pip install google-genai python-dotenv pillow
python scripts/check_setup.py  # Verify setup

Optional: API key rotation for rate limits (set GEMINI_API_KEY_2, GEMINI_API_KEY_3).

Workflow

Step 1: Identify Modality

Input Type	Task	Command
Image (PNG/JPG/WEBP)	Analyze, caption, OCR	--task analyze
Audio (WAV/MP3/AAC)	Transcribe, summarize	--task transcribe
Video (MP4/MOV)	Scene detection, Q&A	--task analyze
PDF/Document	Extract tables, forms	--task extract
Text prompt	Generate image	--task generate
Text prompt	Generate video	--task generate-video

Step 2: Check Limits

• Inline upload: max 20MB
• File API: max 2GB (auto-used for large files)
• Audio transcription: split at 15-minute chunks for full transcript
• Video transcription: extract audio first, then split and transcribe
• Formats: Audio (WAV/MP3/AAC, up to 9.5h), Images (PNG/JPEG/WEBP, up to 3.6k), Video (MP4/MOV, up to 6h), PDF (up to 1k pages)

IF file exceeds limits, use scripts/media_optimizer.py to compress/split first.

Step 3: Execute

Quick check: If gemini CLI is available, use: "<prompt>" | gemini -y -m gemini-2.5-flash

Standard: Use the batch processing script:

# Analyze media
python scripts/gemini_batch_process.py --files <file> --task <analyze|transcribe|extract>

# Generate content
python scripts/gemini_batch_process.py --task generate --prompt "description"
python scripts/gemini_batch_process.py --task generate-video --prompt "description"

Stdin support: cat image.png | python scripts/gemini_batch_process.py --task analyze --prompt "Describe this"

Step 4: Post-Processing

• For transcripts: output in markdown with [HH:MM:SS -> HH:MM:SS] timestamps
• For document extraction: save as structured markdown under docs/assets/
• For generated images/videos: save to working directory with descriptive filename

Step 5: Verification

• Confirm output matches expected format and completeness
• For long transcripts: verify no truncation occurred (check chunk boundaries)
• For generated content: verify quality meets prompt requirements

Models

Purpose	Model	Notes
Analysis (fast)	gemini-2.5-flash	Recommended default
Analysis (advanced)	gemini-2.5-pro	Complex reasoning tasks
Image generation	imagen-4.0-generate-001	Standard quality
Image generation (quality)	imagen-4.0-ultra-generate-001	Best quality
Image generation (speed)	imagen-4.0-fast-generate-001	Fastest
Video generation	veo-3.1-generate-preview	8s clips with audio

Scripts Reference

• gemini_batch_process.py -- CLI orchestrator for all tasks, auto-resolves API keys and models
• media_optimizer.py -- Compress/resize/split media to fit Gemini limits
• document_converter.py -- Convert PDFs/images/Office docs to markdown
• check_setup.py -- Verify environment, dependencies, and API key

Use --help on any script for full options.

Examples

Example 1: Transcribe a Meeting Recording

Input: 45-minute meeting audio file meeting-2025-01-15.mp3

Steps:

1. File is >15min, so split first:

   python scripts/media_optimizer.py --input meeting-2025-01-15.mp3 --split-duration 900
   

2. Transcribe each chunk:

   python scripts/gemini_batch_process.py --files meeting-part-*.mp3 --task transcribe
   

3. Output: Markdown file with timestamps, speaker detection, and metadata (duration, topics covered)

Example 2: Extract Data from a PDF Report

Input: Quarterly HR report PDF with tables, charts, and forms

Steps:

1. Convert and extract:

   python scripts/document_converter.py --input quarterly-report.pdf --output docs/assets/
   

2. Output: Structured markdown with tables preserved, chart descriptions, and form field values extracted

Detailed References

Load for in-depth guidance:

Topic	File
Audio processing	references/audio-processing.md
Vision/image analysis	references/vision-understanding.md
Image generation	references/image-generation.md
Video analysis	references/video-analysis.md
Video generation	references/video-generation.md
Music generation	references/music-generation.md

Related Skills

• ai-artist -- for prompt engineering and optimization (not media processing)
• media-processing -- for FFmpeg-based audio/video encoding without AI
• pdf-to-markdown -- for simple PDF text extraction without vision AI

---

IMPORTANT Task Planning Notes (MUST FOLLOW)

• Always plan and break work into many small todo tasks
• Always add a final review todo task to verify work quality and identify fixes/enhancements