Hermes Vision Setup

Hermes Vision Setup

Hermes can analyze images even when the main chat model is text-only (e.g. deepseek-v4-flash): attached images are pre-described by an auxiliary vision model via vision_analyze, and that description is fed to the main model as text. This skill covers diagnosing, probing, and configuring the auxiliary vision backend. It does NOT cover native vision on vision-capable main models (that needs no setup).

When to Use

• Main model is text-only and image analysis fails, or the image is never described to the model.
• vision_analyze errors, or a turn starts with 👁️ analyzing … ⚠ vision analysis failed.
• After switching provider/model, image analysis stops working.

Prerequisites

• The hermes-agent repo checkout + its venv (for the probe script): cd <repo> && venv/Scripts/python.exe …
• At least one candidate vision-capable model reachable with an existing API key.

How It Works (mechanism)

• agent.image_input_mode: auto (config.yaml) decides per turn: native = pixels go to the main model (only when it supports vision); text = each attached image is pre-analyzed by vision_analyze_tool and the description is prepended to the user message (cli.py::_preprocess_images_with_vision; the gateway does the same). Decision table: agent/image_routing.py::decide_image_input_mode.
• vision_analyze (tools/vision_tools.py) resolves its backend through agent/auxiliary_client.py::resolve_vision_provider_client().
• Auto-detection order (auxiliary.vision.provider: auto): 1) main provider+model IF vision-capable, 2) OpenRouter, 3) Nous Portal, 4) DeepInfra. A text-only main model skips step 1 entirely, so auto lands on OpenRouter/Nous — if those are unhealthy or unauthenticated, nothing resolves.
• Explicit auxiliary.vision.{provider,model,base_url,api_key} overrides the chain — the reliable fix.
• Tool gate: check_vision_requirements(). It only proves a client can be RESOLVED — a resolved client does not prove the model accepts images. Only a real API call proves that.

Procedure

1. Inspect state:
   • hermes config get agent.image_input_mode → keep auto.
   • hermes config get auxiliary.vision → note provider/model/base_url.
2. Gate check (repo root, repo venv): venv/Scripts/python.exe -c "from tools.vision_tools import check_vision_requirements; print(check_vision_requirements())"
3. If False, list candidates from the relays/accounts you have keys for: curl -s -H "Authorization: Bearer $KEY" <base_url>/models (base_url comes from the provider profile, e.g. plugins/model-providers/<name>/__init__.py.)
4. Probe candidates with a REAL image call — scripts/probe_vision_backends.py (see Quick Reference). Pick the first OK.
5. Apply the winner (never hand-edit config.yaml): hermes config set auxiliary.vision.provider <provider> hermes config set auxiliary.vision.model <model>
6. Verify E2E: gate → True, then vision_analyze_tool(image_url=<real png>) → {"success": true, "analysis": …}.

Quick Reference

cd <hermes-agent repo>
venv/Scripts/python.exe scripts/probe_vision_backends.py opencode-go/mimo-v2.5 opencode-zen/gemini-3-flash alibaba/qwen-vl-max
# prints one OK/FAIL line per candidate; exit 0 if any worked

Pitfalls

• Thinking models return content=None on small max_tokens — all budget is consumed by reasoning. Probe with max_tokens ≥ 400, or send extra_body={"thinking": {"type": "disabled"}}.
• Test images must be valid — a hand-built minimal PNG can fail with "Multimodal data is corrupted or cannot be processed". Generate with PIL.
• A relay's /v1/models listing lies — listed models can still be rejected with 400 Unsupported model (subscription-tier gating). Always E2E-probe the actual candidate.
• Client resolves ≠ model accepts images — check_vision_requirements() True is necessary, not sufficient.
• auto never tries your main relay for a text-only main model — by design. Configure auxiliary.vision explicitly.
• Fallback-chain failures are easy to miss — OpenRouter "payment / credit error" and Nous "no auth" lines only appear in logs and mark those providers unhealthy for 60s. Read the aux-client log lines.
• 401 classes mean different fixes: Insufficient balance / CreditsError = top up the account; Incorrect API key = stale/revoked key; Unsupported model = wrong model id for that tier.

Verification

• check_vision_requirements() → True.
• vision_analyze_tool(image_url=<real file>) → {"success": true, "analysis": …} with a sensible answer.
• In-chat: attach an image → CLI prints 👁️ analyzing … ✓ image analyzed, and the text-only model describes the image content.

See references/vision-resolution-chain.md for the full resolution order, failure-mode table, and observed state on this machine.