zinohome Multimodal Understanding
Analyze images via NoDesk AI Gateway. The API is OpenAI Vision-compatible.
Configuration
No setup needed. Scripts read the API key from ~/.deskclaw/nanobot/config.json, checking providers.custom → anthropic → openai (first non-empty key wins).
Workflow
Local file (most common)
When the user provides a local file path (e.g. /Users/.../photo.jpg, ~/Desktop/img.png):
bash <skill_dir>/scripts/multimodal-call.sh --file '<image_path>' '<user_prompt>' [max_tokens] [model]
The script auto-detects the image type and base64-encodes it. No manual conversion needed.
Image URL
When the user provides a public image URL (e.g. https://example.com/photo.jpg):
bash <skill_dir>/scripts/multimodal-call.sh --url '<image_url>' '<user_prompt>' [max_tokens] [model]
The URL must be publicly accessible. No download or base64 conversion needed.
Parameters
max_tokens: default 1000. Use 300 for short descriptions, 2000+ for OCR / detailed analysis.model: defaultgpt-4o.
Present results
Extract choices[0].message.content from the JSON response and show it directly to the user.
Error Handling
| HTTP Status | Meaning | Action |
|---|---|---|
| 401 / 403 | Invalid API key | Ask user to verify their key |
| 429 | Quota exceeded or rate limited | Tell user to check quota on gateway dashboard |
| Timeout | Image too large or gateway busy | Retry once; if still failing, tell user |
Important
- Local file path → use
--filemode. Tell users: instead of pasting/dragging images, type the file path as text for best results. - Image URL → use
--urlmode. URL must be publicly accessible. - Each image costs ~200–800 prompt tokens. Multimodal and LLM chat share the same token quota.