# Hermes Auxiliary Config

> Configure Hermes auxiliary side-LLM backends (vision).

- Skill: `wcpaka-lgtm/hermes-auxiliary-config` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add wcpaka-lgtm/hermes-auxiliary-config`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wcpaka-lgtm/hermes-auxiliary-config/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: wcpaka-lgtm (https://skillmd.com/u/wcpaka-lgtm)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/wcpaka-lgtm/hermes-auxiliary-config

---


# Hermes Auxiliary Model Configuration

Configure the `auxiliary:` section of Hermes config.yaml — the side-LLM tasks that run behind the main agent: `vision` (image analysis via `vision_analyze` + image-input routing), `web_extract`, `compression`, `title_generation`, `skills_hub`, `approval`, `mcp`, `curator`, etc. Covers provider/model selection, the `fallback_chain` mechanism, and how to verify a backend actually works.

## When to Use
- Main model is text-only (e.g. deepseek-v4-flash) and the user asks "can you see images" / image analysis fails.
- An auxiliary task (vision, compression, titles) returns errors like "Insufficient balance", 401/402, "unknown variant `image_url`", or falls back to unusable providers.
- User wants a free/cheap backend for side tasks with automatic fallback when quota runs out.
- Diagnosing which backend an aux task actually resolved to.

## Config Surface (config.yaml)
```yaml
auxiliary:
  vision:            # one block per task
    provider: auto   # "auto" | explicit provider name (opencode-zen, openrouter, google, ...)
    model: ''        # explicit model id (e.g. mimo-v2.5-free)
    base_url: ''     # '' → provider profile's base_url; set for custom endpoints
    api_key: ''      # '' → provider env var
    timeout: 120
    fallback_chain:  # tried when the primary fails at runtime (see semantics below)
      - provider: opencode-go
        model: mimo-v2.5
```
Every entry in `fallback_chain` needs at least `provider`; `model`, `base_url`, `api_key` optional. Entries resolve through the central provider router (registered profiles supply base_url + env key automatically).

## Auto Resolution Chains (`agent/auxiliary_client.py`)
- **Text tasks (auto)**: main provider+model → OpenRouter → Nous Portal → custom endpoint → native Anthropic → direct API-key providers → None.
- **Vision tasks (auto)**: main provider *only if vision-capable* → OpenRouter → Nous → DeepInfra → None. A text-only main model (e.g. DeepSeek V4 Flash) is **skipped**, not tried — the chain falls through. Unknown/custom relays (opencode-* etc.) are not in the auto chain; you must set `auxiliary.vision.provider` explicitly.
- Explicit `provider:` skips auto-detection for that task but keeps the fallback machinery.

## Fallback Semantics (verified in code, 2026-08)
- **Order on runtime failure**: ① user-configured `fallback_chain` (per task) → ② main fallback_providers (auto tasks) → ③ built-in discovery chain → ④ main-agent-model safety net.
- **Quota/payment detection** (`_is_payment_error`): HTTP 402 always; 402/403/404/429/None status + message keywords ("credits", "insufficient funds", "billing", "quota exceeded", "daily limit", "resource exhausted", "weekly usage limit", "reached your session usage limit", ...). The fallback chain block runs for any exception reason, not just payment.
- **10-minute unhealthy cache** (`_AUX_UNHEALTHY_TTL_SECONDS = 600`): after a confirmed payment error a provider is skipped for 10 min, then **auto-retried on the first request after TTL expiry** — a topped-up/free-quota account recovers with zero manual intervention. The cache is in-process only (restart clears it → primary tried immediately again).
- **Switch-back behavior**: every request tries the primary first (primary-first resolution is per-request). So "free quota runs out → fallback serves → quota replenishes → automatically back on the free model" works out of the box.

## PITFALL: `hermes config set` cannot write list values
`set_config_value` only coerces bool/int/float — a list-of-dicts `fallback_chain` passed as a string is stored as a STRING, which `_try_configured_fallback_chain` ignores (`isinstance(chain, list)` check). Scalars are fine via `hermes config set auxiliary.vision.provider <name>`. For the list, write it through the repo's own machinery (identical to the CLI's write path):
```python
from hermes_cli.config import fast_safe_load, get_config_path
from utils import atomic_yaml_write
cfg = fast_safe_load(open(get_config_path(), encoding="utf-8")) or {}
v = cfg.setdefault("auxiliary", {}).setdefault("vision", {})
v["fallback_chain"] = [{"provider": "opencode-go", "model": "mimo-v2.5"}]
atomic_yaml_write(get_config_path(), cfg, sort_keys=False)
```

## Verification Workflow (do this BEFORE telling the user it works)
1. `from tools.vision_tools import check_vision_requirements; check_vision_requirements()` → True means some backend resolves.
2. `from agent.auxiliary_client import resolve_vision_provider_client; p, client, m = resolve_vision_provider_client()` → shows which provider/model the primary resolves to (print p/m, never the client's key).
3. Simulate the fallback: `_try_configured_fallback_chain("vision", "<primary_provider>", reason="payment error")` → should return the fallback client when the chain is wired.
4. **Real E2E image call** (client resolution ≠ vision works): build a PIL image, send a chat.completions call with an `image_url` data-URL part, assert the answer. Run `scripts/probe_vision_backend.py` (support file) for a ready-made probe.
5. Reasoning-model gotcha: a vision call can return `content=None`/`finish=length` when max_tokens is tiny (thinking tokens eat the budget). Retry with max_tokens ≥ 300 before concluding the model can't see images.
6. Note: config is re-read per call (mtime-based cache invalidation) — no restart needed for the vision tool, but a `/new`/restart is the safe fallback if behavior seems stale.

## Known-good backends (this user's machine, verified 2026-08)
- `opencode-zen` + `mimo-v2.5-free` — **free** Zen model, vision-capable, works with zero account balance (paid models on the same key return 401 CreditsError "Insufficient balance"). Verified: correctly identified colors.
- `opencode-go` + `mimo-v2.5` — vision-capable (verified), used as fallback_chain.
- Free Zen models that CANNOT see images (404 "No endpoints that support image input"): ling-3.0-flash-free, nemotron-3-ultra-free, north-mini-code-free, laguna-s-2.1-free; text-only (400 unknown variant): deepseek-v4-flash-free, big-pickle; longcat-2.0-free accepts images but vision is unreliable.
- DashScope intl (`alibaba`, dashscope-intl.aliyuncs.com) — key may be invalid (401) even when set in .env; test before relying on it.
- Testing procedure: query `GET <relay>/v1/models` with the bearer key to enumerate available model ids before guessing.

## How Attached Images Reach a Text-Only Main Model
`agent.image_input_mode: auto` (config) → `decide_image_input_mode(provider, model, cfg)`: "native" when the main model reports vision support, else "text" → the CLI/gateway pre-runs `vision_analyze` per image and prepends the description to the user message (path is included so the agent can re-inspect with `vision_analyze`). The main model never sees pixels — analysis quality is bounded by the aux vision model's description.

## Pitfalls
- Never assume a provider in `.env` works — 401 (wrong/expired key) and 401-CreditsError (no balance) both look similar; the message tells them apart.
- `hermes config set` echoes values; never pass API keys on the CLI (they'd land in shell history) — keys live in `.env` via `hermes setup`/`hermes tools`, config holds only provider/model names.
- Tool search on Windows sometimes fails on absolute paths with forward slashes — use relative paths from the repo root or terminal cd + grep.
- The unhealthy-cache warning log line ("marking X unhealthy for 60s") also appears for providers that never had a key (openrouter without balance, nous without `hermes auth`) — it is noise from the auto chain, not proof the primary is broken.