菜单
Vision, audio, video generation, and multimodal LLM integration patterns. Use when processing images, transcribing audio, generating speech, generating AI video (Kling v3, Sora 2, Veo 3.1 std/lite/fast, Runway Gen-4.5 via `gen4_turbo`), or building multimodal AI pipelines.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/yonatangross/orchestkit --skill multimodal-llm命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | multimodal-llm |
| license | MIT |
| compatibility | Claude Code 2.1.220+. |
| author | OrchestKit |
| version | 2.1.1 |
| description | Vision, audio, video generation, and multimodal LLM integration patterns. Use when processing images, transcribing audio, generating speech, generating AI video (Kling v3, Sora 2, Veo 3.1 std/lite/fast, Runway Gen-4.5 via `gen4_turbo`), or building multimodal AI pipelines. |
| tags | ["vision","audio","video","multimodal","image","speech","transcription","tts","kling","sora","veo","video-generation"] |
| user-invocable | false |
| disable-model-invocation | true |
| context | fork |
| agent | multimodal-specialist |
| complexity | high |
| persuasion-type | reference |
| effort | high |
| metadata | {"category":"mcp-enhancement"} |
| allowed-tools | ["Read","Glob","Grep","WebFetch","WebSearch"] |
Integrate vision, audio, and video generation capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, text-to-speech, and AI video generation (Kling v3, Sora 2, Veo 3.1 std/lite/fast tiers, Runway Gen-4.5 via gen4_turbo).
Canonical model IDs (pinned against
yonatan-hq/platform/apps/api/app/config.py):
Provider Model IDs Anthropic claude-opus-5(recommended, 2,576 px budget, production default),claude-opus-4-8,claude-opus-4-7,claude-opus-4-6,claude-sonnet-4-6,claude-haiku-4-5-20251001.claude-fable-5is Anthropic's frontier tier above Opus (GA 2026-07). Premium cost — never auto-pin it; the fable-spend-consent gate requires explicit user consent before any Fable spendOpenAI gpt-5.5(current flagship)gemini-3.1-pro-preview(flagship),gemini-3.1-flash-lite-preview(cost)Veo veo-3.1-generate-preview/veo-3.1-lite-generate-preview/veo-3.1-fast-generate-previewKling kling-v3(model_name field in Kling API)Runway gen4_turbo(product label: Gen-4.5)
| Category | Rules | Impact | When to Use |
|---|---|---|---|
| Vision: Image Analysis | 1 | HIGH | Image captioning, VQA, multi-image comparison, object detection |
| Vision: Document Understanding | 1 | HIGH | OCR, chart/diagram analysis, PDF processing, table extraction |
| Vision: Model Selection | 1 | MEDIUM |
| Choosing provider, cost optimization, image size limits |
| Audio: Speech-to-Text | 1 | HIGH | Transcription, speaker diarization, long-form audio |
| Audio: Text-to-Speech | 1 | MEDIUM | Voice synthesis, expressive TTS, multi-speaker dialogue |
| Audio: Model Selection | 1 | MEDIUM | Real-time voice agents, provider comparison, pricing |
| Video: Model Selection | 1 | HIGH | Choosing video gen provider (Kling, Sora, Veo, Runway) |
| Video: API Patterns | 1 | HIGH | Async task polling, SDK integration, webhook callbacks |
| Video: Multi-Shot | 1 | HIGH | Storyboarding, character elements, scene consistency |
Total: 9 rules across 3 categories (Vision, Audio, Video Generation)
Send images to multimodal LLMs for captioning, visual QA, and object detection. Always set max_tokens and resize images before encoding.
| Rule | File | Key Pattern |
|---|---|---|
| Image Analysis | rules/vision-image-analysis.md | Base64 encoding, multi-image, bounding boxes |
Extract structured data from documents, charts, and PDFs using vision models.
| Rule | File | Key Pattern |
|---|---|---|
| Document Vision | rules/vision-document.md | PDF page ranges, detail levels, OCR strategies |
Choose the right vision provider based on accuracy, cost, and context window needs.
| Rule | File | Key Pattern |
|---|---|---|
| Vision Models | rules/vision-models.md | Provider comparison, token costs, image limits |
Convert audio to text with speaker diarization, timestamps, and sentiment analysis.
| Rule | File | Key Pattern |
|---|---|---|
| Speech-to-Text | rules/audio-speech-to-text.md | Gemini long-form, GPT-4o-Transcribe, AssemblyAI features |
Generate natural speech from text with voice selection and expressive cues.
| Rule | File | Key Pattern |
|---|---|---|
| Text-to-Speech | rules/audio-text-to-speech.md | Gemini TTS, voice config, auditory cues |
Select the right audio/voice provider for real-time, transcription, or TTS use cases.
| Rule | File | Key Pattern |
|---|---|---|
| Audio Models | rules/audio-models.md | Real-time voice comparison, STT benchmarks, pricing |
Choose the right video generation provider based on use case, duration, and budget.
| Rule | File | Key Pattern |
|---|---|---|
| Video Models | rules/video-generation-models.md | Kling vs Sora vs Veo vs Runway, pricing, capabilities |
Integrate video generation APIs with proper async polling, SDKs, and webhook callbacks.
| Rule | File | Key Pattern |
|---|---|---|
| API Integration | rules/video-generation-patterns.md | Kling REST, fal.ai SDK, Vercel AI SDK, task polling |
Generate multi-scene videos with consistent characters using storyboarding and character elements.
| Rule | File | Key Pattern |
|---|---|---|
| Multi-Shot | rules/video-multi-shot.md | Kling v3 character elements, 6-shot storyboards, identity binding |
| Decision | Recommendation |
|---|---|
| High accuracy vision | claude-opus-5 (production default, 2,576 px vision budget, 3× what Opus 4.6 allotted; give it crop/analyze tools rather than more thinking, which is the cheaper lever on this model). (claude-fable-5 is the frontier SOTA option, GA 2026-07 — premium cost, use only with explicit consent via the fable-spend-consent gate) |
| Long documents | gemini-3.1-pro-preview (1M+ context) |
| Cost-efficient vision | gemini-3.1-flash-lite-preview (replaces Gemini 2.5 Flash, deprecates Oct 2026) |
| Video analysis | gemini-3.1-pro-preview (native video, supersedes 2.5 Pro) |
| Voice assistant | Grok Voice Agent on Grok 4.20 (fastest, <1s) |
| Emotional voice AI | Gemini Live API |
| Long audio transcription | gemini-3.1-pro-preview (9.5hr) |
| Speaker diarization | AssemblyAI or Gemini |
| Self-hosted STT | Whisper Large V3 |
| Character-consistent video | kling-v3 (Character Elements 3.0) |
| Narrative video / storytelling | Sora 2 (best cause-and-effect coherence) |
| Cinematic B-roll | veo-3.1-generate-preview (camera control + polished motion) |
| Budget drafts | veo-3.1-lite-generate-preview (~$0.05/s, 720/1080p) |
| Mid-tier fast renders | veo-3.1-fast-generate-preview |
| Professional VFX | Runway gen4_turbo (Act-Two motion transfer) |
| High-volume social video | kling-v3 Standard (~$0.20/video) |
| Open-source video gen | Wan 2.6 or LTX-2 |
| Lip-sync / avatar video | kling-v3 (native lip-sync API) |
import anthropic, base64
client = anthropic.Anthropic()
with open("image.png", "rb") as f:
b64 = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.messages.create(
model="claude-opus-5",
max_tokens=1024,
messages=[{"role": "user", "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
{"type": "text", "text": "Describe this image"}
]}]
)
max_tokens on vision requests (responses truncated)high detail level for simple yes/no classificationork:rag-retrieval - Multimodal RAG with image + text retrievalork:llm-integration - General LLM function calling patternsstreaming-api-patterns - WebSocket patterns for real-time audioork:demo-producer - Terminal demo videos (VHS, asciinema) — not AI video genChain patterns for multi-phase pipelines: MCP detection, handoff files, checkpoint-resume, worktree agents, CronCreate monitoring. Use when building or debugging a pipeline skill.
OrchestKit doctor for health diagnostics across manifest integrity, hook configuration, skill validation, agent frontmatter, MCP server connectivity, CC version compatibility, and permission rules. Reports issues with severity levels and auto-remediation suggestions. Validates component counts, detects orphaned entries, and checks CC version matrix compliance. Use when diagnosing plugin health, troubleshooting configuration issues, or running pre-release checks.
Inspects the OrchestKit telemetry pipeline for the current project — lists all known telemetry files with write counts, sizes, schema status, growth trend, and orphan detection. Use when verifying the observability pipeline is healthy, debugging a missing writer, or auditing which files have schema locks vs. which are drift-vulnerable. Read-only — never modifies telemetry files.