# Hermes Vision Setup

> Set up or fix Hermes image analysis for text-only models.

- Skill: `wcpaka-lgtm/hermes-vision-setup` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add wcpaka-lgtm/hermes-vision-setup`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wcpaka-lgtm/hermes-vision-setup/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: wcpaka-lgtm (https://skillmd.com/u/wcpaka-lgtm)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/wcpaka-lgtm/hermes-vision-setup

---


# Hermes Vision Setup

Hermes can analyze images even when the main chat model is text-only (e.g. deepseek-v4-flash): attached images are pre-described by an auxiliary vision model via `vision_analyze`, and that description is fed to the main model as text. This skill covers diagnosing, probing, and configuring the auxiliary vision backend. It does NOT cover native vision on vision-capable main models (that needs no setup).

## When to Use

- Main model is text-only and image analysis fails, or the image is never described to the model.
- `vision_analyze` errors, or a turn starts with `👁️ analyzing … ⚠ vision analysis failed`.
- After switching provider/model, image analysis stops working.

## Prerequisites

- The hermes-agent repo checkout + its venv (for the probe script): `cd <repo> && venv/Scripts/python.exe …`
- At least one candidate vision-capable model reachable with an existing API key.

## How It Works (mechanism)

- `agent.image_input_mode: auto` (config.yaml) decides per turn: `native` = pixels go to the main model (only when it supports vision); `text` = each attached image is pre-analyzed by `vision_analyze_tool` and the description is prepended to the user message (`cli.py::_preprocess_images_with_vision`; the gateway does the same). Decision table: `agent/image_routing.py::decide_image_input_mode`.
- `vision_analyze` (tools/vision_tools.py) resolves its backend through `agent/auxiliary_client.py::resolve_vision_provider_client()`.
- Auto-detection order (`auxiliary.vision.provider: auto`): 1) main provider+model IF vision-capable, 2) OpenRouter, 3) Nous Portal, 4) DeepInfra. A text-only main model skips step 1 entirely, so `auto` lands on OpenRouter/Nous — if those are unhealthy or unauthenticated, nothing resolves.
- Explicit `auxiliary.vision.{provider,model,base_url,api_key}` overrides the chain — the reliable fix.
- Tool gate: `check_vision_requirements()`. It only proves a client can be RESOLVED — a resolved client does not prove the model accepts images. Only a real API call proves that.

## Procedure

1. Inspect state:
   - `hermes config get agent.image_input_mode` → keep `auto`.
   - `hermes config get auxiliary.vision` → note provider/model/base_url.
2. Gate check (repo root, repo venv):
   `venv/Scripts/python.exe -c "from tools.vision_tools import check_vision_requirements; print(check_vision_requirements())"`
3. If False, list candidates from the relays/accounts you have keys for:
   `curl -s -H "Authorization: Bearer $KEY" <base_url>/models`
   (base_url comes from the provider profile, e.g. `plugins/model-providers/<name>/__init__.py`.)
4. Probe candidates with a REAL image call — `scripts/probe_vision_backends.py` (see Quick Reference). Pick the first OK.
5. Apply the winner (never hand-edit config.yaml):
   `hermes config set auxiliary.vision.provider <provider>`
   `hermes config set auxiliary.vision.model <model>`
6. Verify E2E: gate → True, then `vision_analyze_tool(image_url=<real png>)` → `{"success": true, "analysis": …}`.

## Quick Reference

```bash
cd <hermes-agent repo>
venv/Scripts/python.exe scripts/probe_vision_backends.py opencode-go/mimo-v2.5 opencode-zen/gemini-3-flash alibaba/qwen-vl-max
# prints one OK/FAIL line per candidate; exit 0 if any worked
```

## Pitfalls

- **Thinking models return content=None on small max_tokens** — all budget is consumed by reasoning. Probe with max_tokens ≥ 400, or send `extra_body={"thinking": {"type": "disabled"}}`.
- **Test images must be valid** — a hand-built minimal PNG can fail with "Multimodal data is corrupted or cannot be processed". Generate with PIL.
- **A relay's /v1/models listing lies** — listed models can still be rejected with `400 Unsupported model` (subscription-tier gating). Always E2E-probe the actual candidate.
- **Client resolves ≠ model accepts images** — `check_vision_requirements()` True is necessary, not sufficient.
- **`auto` never tries your main relay for a text-only main model** — by design. Configure `auxiliary.vision` explicitly.
- **Fallback-chain failures are easy to miss** — OpenRouter "payment / credit error" and Nous "no auth" lines only appear in logs and mark those providers unhealthy for 60s. Read the aux-client log lines.
- **401 classes mean different fixes**: `Insufficient balance` / `CreditsError` = top up the account; `Incorrect API key` = stale/revoked key; `Unsupported model` = wrong model id for that tier.

## Verification

- `check_vision_requirements()` → True.
- `vision_analyze_tool(image_url=<real file>)` → `{"success": true, "analysis": …}` with a sensible answer.
- In-chat: attach an image → CLI prints `👁️ analyzing … ✓ image analyzed`, and the text-only model describes the image content.

See `references/vision-resolution-chain.md` for the full resolution order, failure-mode table, and observed state on this machine.