Hermes Auxiliary Models

Hermes Auxiliary Models Skill

Hermes routes side-LLM work (vision analysis, context compression, web extraction, title generation, session search) through auxiliary tasks configured under auxiliary: in config.yaml. Each task (vision, compression, web_extract, …) has its own provider/model/base_url/api_key/timeout and an optional fallback_chain. This skill covers configuring, E2E-verifying, and fallback-wiring those backends — most commonly making a text-only main model "see" images.

When to Use

• Main model is text-only (e.g. deepseek-v4-flash) and the user wants image analysis.
• vision_analyze fails, check_vision_requirements() returns False, or image attachments come back as "[image couldn't be analyzed]".
• Setting up automatic fallback for when a free model's quota is exhausted.
• Choosing/verifying a vision-capable model on a relay the user already has a key for.

Prerequisites

• Working hermes install; user config at $HERMES_HOME/config.yaml.
• For E2E tests: the repo venv (venv/Scripts/python.exe on Windows) with Pillow + openai (both ship in the Hermes venv).
• To run the repo test suite: venv needs pytest + pytest-asyncio (pip install pytest pytest-asyncio) — scripts/run_tests.sh silently skips venvs without pytest.

How to Run

1. Understand the routing first

agent/image_routing.py::decide_image_input_mode(provider, model, config) decides how user-attached images reach the main model (agent.image_input_mode: auto|native|text):

• native — pixels attached directly (vision-capable main models).
• text — each image is pre-analyzed via vision_analyze_tool (auxiliary vision model) and the description is prepended as text. Automatic for non-vision models; the vision_analyze tool stays available in every session.

2. E2E-test candidates BEFORE configuring (never configure blind)

1. List the relay's models: curl -H "Authorization: Bearer $KEY" <base_url>/models (e.g. https://opencode.ai/zen/v1/models).
2. Generate a REAL test image with Pillow — hand-built base64 PNGs get rejected ("Multimodal data is corrupted").
3. Resolve + call: resolve_vision_provider_client(provider=..., model=...) from agent/auxiliary_client.py, then client.chat.completions.create(...) with an image_url data part.
4. Ask a one-word question ("What color is this image?"). Use max_tokens ≥ 300 — thinking models burn small budgets and return content='' with finish=length, which looks like failure but means it worked.

3. Configure

hermes config set auxiliary.vision.provider <provider>
hermes config set auxiliary.vision.model <model>

4. Add automatic fallback (free quota exhaustion)

auxiliary.<task>.fallback_chain is consulted on payment/quota/rate-limit errors before the built-in discovery chain. hermes config set only stores scalars (bool/int/float) — a list-of-dicts must be written with the repo's own writer:

from hermes_cli.config import fast_safe_load, get_config_path
from utils import atomic_yaml_write
path = get_config_path()
cfg = fast_safe_load(open(path, encoding='utf-8')) or {}
vision = cfg.setdefault('auxiliary', {}).setdefault('vision', {})
vision['fallback_chain'] = [{'provider': 'opencode-go', 'model': 'mimo-v2.5'}]
atomic_yaml_write(path, cfg, sort_keys=False)

Entry fields: provider (required), model, base_url, api_key, api_mode optional. Entries resolve through the central provider router, so registered profiles (env key + base_url) work without inline values.

5. Verify

• check_vision_requirements() from tools/vision_tools.py → True.
• resolve_vision_provider_client() → returns the configured provider+model with a client.
• Simulate fallback: _try_configured_fallback_chain('vision', '<primary_provider>', reason='payment error') → returns the fallback client.
• Real call: vision_analyze_tool(image_url=<file>, user_prompt=...) → success: true with an accurate description.
• Repo tests: scripts/run_tests.sh tests/tools/test_vision_tools.py -q.

Pitfalls

• 'NoneType' object has no attribute 'strip' or content='' + finish=length = thinking model consumed the max_tokens budget. Raise max_tokens; do not assume failure.
• HTTP 400 "Multimodal data is corrupted" with a tiny hand-built PNG = bad test image, not a bad backend. Use Pillow.
• HTTP 401 "CreditsError: Insufficient balance" on a relay = paid model. Try a -free SKU or another backend; free SKUs serve at zero balance.
• hermes config set cannot write lists/dicts — a string fallback_chain is silently ignored at runtime (isinstance(chain, list) check). Use the atomic_yaml_write path above.
• run_tests.sh reporting "no virtualenv with pytest found" is a setup gap (venv lacks pytest), not a test failure. Install pytest + pytest-asyncio and rerun.
• Async-marked repo tests fail en masse at collection ("Unknown pytest.mark.asyncio") when pytest-asyncio is missing — install it before judging failures.

Verification

• Real image through vision_analyze_tool returns success: true with an accurate description.
• scripts/run_tests.sh tests/tools/test_vision_tools.py -q → all pass.

References

• references/opencode-zen-vision-models.md — OpenCode Zen/Go relay catalog: free vs paid models, which ones see images, base URLs, key names, measured results, and known dead ends.