Hermes Auxiliary Config

Hermes Auxiliary Model Configuration

Configure the auxiliary: section of Hermes config.yaml — the side-LLM tasks that run behind the main agent: vision (image analysis via vision_analyze + image-input routing), web_extract, compression, title_generation, skills_hub, approval, mcp, curator, etc. Covers provider/model selection, the fallback_chain mechanism, and how to verify a backend actually works.

When to Use

• Main model is text-only (e.g. deepseek-v4-flash) and the user asks "can you see images" / image analysis fails.
• An auxiliary task (vision, compression, titles) returns errors like "Insufficient balance", 401/402, "unknown variant image_url", or falls back to unusable providers.
• User wants a free/cheap backend for side tasks with automatic fallback when quota runs out.
• Diagnosing which backend an aux task actually resolved to.

Config Surface (config.yaml)

auxiliary:
  vision:            # one block per task
    provider: auto   # "auto" | explicit provider name (opencode-zen, openrouter, google, ...)
    model: ''        # explicit model id (e.g. mimo-v2.5-free)
    base_url: ''     # '' → provider profile's base_url; set for custom endpoints
    api_key: ''      # '' → provider env var
    timeout: 120
    fallback_chain:  # tried when the primary fails at runtime (see semantics below)
      - provider: opencode-go
        model: mimo-v2.5

Every entry in fallback_chain needs at least provider; model, base_url, api_key optional. Entries resolve through the central provider router (registered profiles supply base_url + env key automatically).

Auto Resolution Chains (agent/auxiliary_client.py)

• Text tasks (auto): main provider+model → OpenRouter → Nous Portal → custom endpoint → native Anthropic → direct API-key providers → None.
• Vision tasks (auto): main provider only if vision-capable → OpenRouter → Nous → DeepInfra → None. A text-only main model (e.g. DeepSeek V4 Flash) is skipped, not tried — the chain falls through. Unknown/custom relays (opencode-* etc.) are not in the auto chain; you must set auxiliary.vision.provider explicitly.
• Explicit provider: skips auto-detection for that task but keeps the fallback machinery.

Fallback Semantics (verified in code, 2026-08)

• Order on runtime failure: ① user-configured fallback_chain (per task) → ② main fallback_providers (auto tasks) → ③ built-in discovery chain → ④ main-agent-model safety net.
• Quota/payment detection (_is_payment_error): HTTP 402 always; 402/403/404/429/None status + message keywords ("credits", "insufficient funds", "billing", "quota exceeded", "daily limit", "resource exhausted", "weekly usage limit", "reached your session usage limit", ...). The fallback chain block runs for any exception reason, not just payment.
• 10-minute unhealthy cache (_AUX_UNHEALTHY_TTL_SECONDS = 600): after a confirmed payment error a provider is skipped for 10 min, then auto-retried on the first request after TTL expiry — a topped-up/free-quota account recovers with zero manual intervention. The cache is in-process only (restart clears it → primary tried immediately again).
• Switch-back behavior: every request tries the primary first (primary-first resolution is per-request). So "free quota runs out → fallback serves → quota replenishes → automatically back on the free model" works out of the box.

PITFALL: hermes config set cannot write list values

set_config_value only coerces bool/int/float — a list-of-dicts fallback_chain passed as a string is stored as a STRING, which _try_configured_fallback_chain ignores (isinstance(chain, list) check). Scalars are fine via hermes config set auxiliary.vision.provider <name>. For the list, write it through the repo's own machinery (identical to the CLI's write path):

from hermes_cli.config import fast_safe_load, get_config_path
from utils import atomic_yaml_write
cfg = fast_safe_load(open(get_config_path(), encoding="utf-8")) or {}
v = cfg.setdefault("auxiliary", {}).setdefault("vision", {})
v["fallback_chain"] = [{"provider": "opencode-go", "model": "mimo-v2.5"}]
atomic_yaml_write(get_config_path(), cfg, sort_keys=False)

Verification Workflow (do this BEFORE telling the user it works)

1. from tools.vision_tools import check_vision_requirements; check_vision_requirements() → True means some backend resolves.
2. from agent.auxiliary_client import resolve_vision_provider_client; p, client, m = resolve_vision_provider_client() → shows which provider/model the primary resolves to (print p/m, never the client's key).
3. Simulate the fallback: _try_configured_fallback_chain("vision", "<primary_provider>", reason="payment error") → should return the fallback client when the chain is wired.
4. Real E2E image call (client resolution ≠ vision works): build a PIL image, send a chat.completions call with an image_url data-URL part, assert the answer. Run scripts/probe_vision_backend.py (support file) for a ready-made probe.
5. Reasoning-model gotcha: a vision call can return content=None/finish=length when max_tokens is tiny (thinking tokens eat the budget). Retry with max_tokens ≥ 300 before concluding the model can't see images.
6. Note: config is re-read per call (mtime-based cache invalidation) — no restart needed for the vision tool, but a /new/restart is the safe fallback if behavior seems stale.

Known-good backends (this user's machine, verified 2026-08)

• opencode-zen + mimo-v2.5-free — free Zen model, vision-capable, works with zero account balance (paid models on the same key return 401 CreditsError "Insufficient balance"). Verified: correctly identified colors.
• opencode-go + mimo-v2.5 — vision-capable (verified), used as fallback_chain.
• Free Zen models that CANNOT see images (404 "No endpoints that support image input"): ling-3.0-flash-free, nemotron-3-ultra-free, north-mini-code-free, laguna-s-2.1-free; text-only (400 unknown variant): deepseek-v4-flash-free, big-pickle; longcat-2.0-free accepts images but vision is unreliable.
• DashScope intl (alibaba, dashscope-intl.aliyuncs.com) — key may be invalid (401) even when set in .env; test before relying on it.
• Testing procedure: query GET <relay>/v1/models with the bearer key to enumerate available model ids before guessing.

How Attached Images Reach a Text-Only Main Model

agent.image_input_mode: auto (config) → decide_image_input_mode(provider, model, cfg): "native" when the main model reports vision support, else "text" → the CLI/gateway pre-runs vision_analyze per image and prepends the description to the user message (path is included so the agent can re-inspect with vision_analyze). The main model never sees pixels — analysis quality is bounded by the aux vision model's description.

Pitfalls

• Never assume a provider in .env works — 401 (wrong/expired key) and 401-CreditsError (no balance) both look similar; the message tells them apart.
• hermes config set echoes values; never pass API keys on the CLI (they'd land in shell history) — keys live in .env via hermes setup/hermes tools, config holds only provider/model names.
• Tool search on Windows sometimes fails on absolute paths with forward slashes — use relative paths from the repo root or terminal cd + grep.
• The unhealthy-cache warning log line ("marking X unhealthy for 60s") also appears for providers that never had a key (openrouter without balance, nous without hermes auth) — it is noise from the auto chain, not proof the primary is broken.