| name | multimodal-understanding |
| description | Analyze images and visual content using the NoDesk AI Gateway multimodal API. ALWAYS use this skill when the user mentions an image file path, image URL, or asks to analyze/describe any image. Do NOT use the built-in image tool — always route through this skill for better accuracy. |
| slug | multimodal-understanding |
| version | 1.1.8 |
| displayName | 图像识别（Multimodal Understanding） |
| summary | 识别与理解图片内容，支持本地文件和 URL 输入。 |
| tags | image, vision, multimodal, understanding |
Multimodal Understanding
Analyze images via NoDesk AI Gateway. The API is OpenAI Vision-compatible.
Configuration
No setup needed. Scripts read the API key from ~/.deskclaw/nanobot/config.json, checking providers.custom → anthropic → openai (first non-empty key wins).
Workflow
Local file (most common)
When the user provides a local file path (e.g. /Users/.../photo.jpg, ~/Desktop/img.png):
bash <skill_dir>/scripts/multimodal-call.sh --file '<image_path>' '<user_prompt>' [max_tokens] [model]
The script auto-detects the image type and base64-encodes it. No manual conversion needed.
Image URL
When the user provides a public image URL (e.g. https://example.com/photo.jpg):
bash <skill_dir>/scripts/multimodal-call.sh --url '<image_url>' '<user_prompt>' [max_tokens] [model]
The URL must be publicly accessible. No download or base64 conversion needed.
Parameters
max_tokens: default 1000. Use 300 for short descriptions, 2000+ for OCR / detailed analysis.model: default gpt-4o.
Present results
Extract choices[0].message.content from the JSON response and show it directly to the user.
Error Handling
| HTTP Status | Meaning | Action |
|---|
| 401 / 403 | Invalid API key | Ask user to verify their key |
| 429 | Quota exceeded or rate limited | Tell user to check quota on gateway dashboard |
| Timeout | Image too large or gateway busy | Retry once; if still failing, tell user |
Important
- Local file path → use
--file mode. Tell users: instead of pasting/dragging images, type the file path as text for best results.
- Image URL → use
--url mode. URL must be publicly accessible.
- Each image costs ~200–800 prompt tokens. Multimodal and LLM chat share the same token quota.
