# Hermes Auxiliary Models

> Set up Hermes auxiliary LLM backends (vision, fallbacks).

- Skill: `wcpaka-lgtm/hermes-auxiliary-models` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add wcpaka-lgtm/hermes-auxiliary-models`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wcpaka-lgtm/hermes-auxiliary-models/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: wcpaka-lgtm (https://skillmd.com/u/wcpaka-lgtm)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/wcpaka-lgtm/hermes-auxiliary-models

---


# Hermes Auxiliary Models Skill

Hermes routes side-LLM work (vision analysis, context compression, web extraction, title generation, session search) through *auxiliary* tasks configured under `auxiliary:` in config.yaml. Each task (`vision`, `compression`, `web_extract`, …) has its own provider/model/base_url/api_key/timeout and an optional `fallback_chain`. This skill covers configuring, E2E-verifying, and fallback-wiring those backends — most commonly making a text-only main model "see" images.

## When to Use

- Main model is text-only (e.g. deepseek-v4-flash) and the user wants image analysis.
- `vision_analyze` fails, `check_vision_requirements()` returns False, or image attachments come back as "[image couldn't be analyzed]".
- Setting up automatic fallback for when a free model's quota is exhausted.
- Choosing/verifying a vision-capable model on a relay the user already has a key for.

## Prerequisites

- Working `hermes` install; user config at `$HERMES_HOME/config.yaml`.
- For E2E tests: the repo venv (`venv/Scripts/python.exe` on Windows) with `Pillow` + `openai` (both ship in the Hermes venv).
- To run the repo test suite: venv needs `pytest` + `pytest-asyncio` (`pip install pytest pytest-asyncio`) — `scripts/run_tests.sh` silently skips venvs without pytest.

## How to Run

### 1. Understand the routing first

`agent/image_routing.py::decide_image_input_mode(provider, model, config)` decides how user-attached images reach the main model (`agent.image_input_mode: auto|native|text`):

- `native` — pixels attached directly (vision-capable main models).
- `text` — each image is pre-analyzed via `vision_analyze_tool` (auxiliary vision model) and the description is prepended as text. Automatic for non-vision models; the `vision_analyze` tool stays available in every session.

### 2. E2E-test candidates BEFORE configuring (never configure blind)

1. List the relay's models: `curl -H "Authorization: Bearer $KEY" <base_url>/models` (e.g. `https://opencode.ai/zen/v1/models`).
2. Generate a REAL test image with Pillow — hand-built base64 PNGs get rejected ("Multimodal data is corrupted").
3. Resolve + call: `resolve_vision_provider_client(provider=..., model=...)` from `agent/auxiliary_client.py`, then `client.chat.completions.create(...)` with an `image_url` data part.
4. Ask a one-word question ("What color is this image?"). Use `max_tokens` ≥ 300 — thinking models burn small budgets and return `content=''` with `finish=length`, which looks like failure but means it worked.

### 3. Configure

```bash
hermes config set auxiliary.vision.provider <provider>
hermes config set auxiliary.vision.model <model>
```

### 4. Add automatic fallback (free quota exhaustion)

`auxiliary.<task>.fallback_chain` is consulted on payment/quota/rate-limit errors before the built-in discovery chain. **`hermes config set` only stores scalars (bool/int/float) — a list-of-dicts must be written with the repo's own writer**:

```python
from hermes_cli.config import fast_safe_load, get_config_path
from utils import atomic_yaml_write
path = get_config_path()
cfg = fast_safe_load(open(path, encoding='utf-8')) or {}
vision = cfg.setdefault('auxiliary', {}).setdefault('vision', {})
vision['fallback_chain'] = [{'provider': 'opencode-go', 'model': 'mimo-v2.5'}]
atomic_yaml_write(path, cfg, sort_keys=False)
```

Entry fields: `provider` (required), `model`, `base_url`, `api_key`, `api_mode` optional. Entries resolve through the central provider router, so registered profiles (env key + base_url) work without inline values.

### 5. Verify

- `check_vision_requirements()` from `tools/vision_tools.py` → True.
- `resolve_vision_provider_client()` → returns the configured provider+model with a client.
- Simulate fallback: `_try_configured_fallback_chain('vision', '<primary_provider>', reason='payment error')` → returns the fallback client.
- Real call: `vision_analyze_tool(image_url=<file>, user_prompt=...)` → `success: true` with an accurate description.
- Repo tests: `scripts/run_tests.sh tests/tools/test_vision_tools.py -q`.

## Pitfalls

- `'NoneType' object has no attribute 'strip'` or `content=''` + `finish=length` = thinking model consumed the max_tokens budget. Raise max_tokens; do not assume failure.
- HTTP 400 "Multimodal data is corrupted" with a tiny hand-built PNG = bad test image, not a bad backend. Use Pillow.
- HTTP 401 "CreditsError: Insufficient balance" on a relay = paid model. Try a `-free` SKU or another backend; free SKUs serve at zero balance.
- `hermes config set` cannot write lists/dicts — a string fallback_chain is silently ignored at runtime (`isinstance(chain, list)` check). Use the atomic_yaml_write path above.
- `run_tests.sh` reporting "no virtualenv with pytest found" is a setup gap (venv lacks pytest), not a test failure. Install `pytest` + `pytest-asyncio` and rerun.
- Async-marked repo tests fail en masse at collection ("Unknown pytest.mark.asyncio") when pytest-asyncio is missing — install it before judging failures.

## Verification

- Real image through `vision_analyze_tool` returns `success: true` with an accurate description.
- `scripts/run_tests.sh tests/tools/test_vision_tools.py -q` → all pass.

## References

- `references/opencode-zen-vision-models.md` — OpenCode Zen/Go relay catalog: free vs paid models, which ones see images, base URLs, key names, measured results, and known dead ends.