Hermes Gateway Image Routing

Hermes Gateway Image Routing

Diagnose and fix how the Hermes Telegram gateway routes images to LLM models.

Architecture

When a user sends an image via Telegram, the gateway decides routing in gateway/run.py:

Architectural Classification: PRMT (Pre-Routing Modality Translation)

Hermes' image routing does NOT fit into the standard multimodal router taxonomy found in the literature:

PRMT key properties:

• Fault isolation: Vision failure → graceful "sorry, can't see" — no cascade crash, no model swap
• No 413 risk: Image bytes never enter reasoning context — transcript text only (zero risk of oversized payload)
• Provider-agnostic fallback: Every fallback model can process text transcripts — none need vision capability
• Auditability: Vision transcript can be inspected before reasoning begins
• Tradeoff: Translation errors are unrecoverable — if Qwen-VL misses detail, DeepSeek cannot recover (it never saw original pixels)

The standard research (vLLM Semantic Router, RouteLLM, MMR-Bench, NVIDIA LLM Router v2) assumes the reasoning model must handle all modalities natively. PRMT reverses this: the reasoning model never needs to know an image existed. Vision processing finishes at the gateway.

See references/prmt-architecture.md for full analysis, taxonomy comparison, and related research.

Path: NATIVE

The image bytes are attached as image_url content parts directly into the user message. The active model sees pixels natively.

• Condition: The active model supports vision (Claude, GPT-4o, Qwen-VL, Gemini, MiniMax-M3)
• How to verify: _decide_image_input_mode() returns "native"

Path: TEXT + [IMAGE TRANSCRIPT] (current, preferred)

When the primary model is text-only (e.g. DeepSeek V4 Flash), the gateway calls _enrich_message_with_vision() which:

1. Runs vision_analyze_tool against the configured auxiliary.vision provider
2. Currently: Qwen-VL-Max via MuleRouter (qwen-vl-max)
3. Produces a structured [IMAGE TRANSCRIPT] with SCENE/OCR/DATA/IDENTITY sections
4. Prepends it to the user message as plain text
5. The primary model receives TEXT ONLY — zero image bytes in context

Configuration (~/.hermes/config.yaml):

auxiliary:
  vision:
    provider: mulerouter
    model: qwen-vl-max  # same provider + key as primary chat model
    timeout: 120

Key sourced from MULEROUTER_API_KEY in kunci-mas.env — same key as the chat provider. This collapses vision and chat into one auth domain: if the primary provider works, vision works. No split failure domain.

WHY MuleRouter instead of OpenRouter: Previously vision enrichment used OpenRouter (separate key, separate billing). When OpenRouter hit a payment error, enrichment was skipped, raw image bytes hit the text-only primary model, and the entire fallback chain crashed with 413. With MuleRouter, vision shares the same key and provider as chat — a single failure domain means vision doesn't fail independently of the primary model.

Condition: _decide_image_input_mode() returns "text" How to verify: Message contains [IMAGE TRANSCRIPT — generated by vision model, not seen by agent directly] Log: Gateway log shows "Image routing: enrich via vision_analyze. N image(s) will be analyzed."

Why this is preferred over model swap:

• Vision call is isolated — if the auxiliary model fails, agent still gets the message with a fallback note
• Output is TEXT — every fallback chain model can process it (no 413 cascade from image bytes)
• Primary model stays DeepSeek Flash (proven, reliable) — no model swap at runtime
• No single point of failure that poisons all fallbacks

Model Selection: qwen-vl-max via MuleRouter is the default

PRMT only needs good scene/OCR description — the sweet spot is qwen-vl-max on MuleRouter (same key as chat, no split failure domain):

Design rationale: Sharing the same provider and key for vision and chat eliminates the split failure domain. If OpenRouter had a payment error, vision enrichment failed while chat still worked — causing raw image bytes to hit the text-only primary → 413 cascade. With MuleRouter, if the primary provider works, vision works.

Switch command when you need a different tier:

hermes config set auxiliary.vision.model qwen3-vl-plus
hermes gateway restart

MuleRouter Integration

See references/mulerouter-integration.md for full details.

Status: ✅ Active auxiliary.vision.provider since 2026-07-30. Key registered (MULEROUTER_API_KEY in kunci-mas.env), vision tested and working.

MuleRouter vs OpenRouter for vision:

Why MuleRouter won over OpenRouter for PRMT:

• Single failure domain: vision and chat share one provider + one key. If the key works for chat, it works for vision. No split-brain where chat responds but vision enrichment silently fails.
• Single bill: one provider bill instead of tracking OpenRouter credits + MuleRouter credits.
• Qwen-VL-Max quality: proven sufficient for PRMT — accurate SCENE/OCR/DATA/IDENTITY extraction on Telegram images.

MuleRouter vision test results (2026-07-30):

• qwen-vl-max model via /vendors/openai/v1/chat/completions — text ✅, base64 image ✅
• qwen3-vl-plus — text ✅, base64 image ✅ (faster, lower cost)
• Production use: successfully enriches Telegram images with 4-section [IMAGE TRANSCRIPT] format

Pre-requisites for switching to MuleRouter:

hermes config set auxiliary.vision.provider mulerouter
hermes config set auxiliary.vision.model qwen-vl-max
hermes config set auxiliary.vision.base_url ""
hermes config set auxiliary.vision.api_key ""
# Then regenerate flat.env and restart gateway
# ⚠️  Run this from a DIFFERENT shell, not inside the gateway session:
systemctl restart hermes-asi-gateway

⛔ DEPRECATED: Path B model swap (2026-07-29 to 2026-07-30)

Previously used a model-override approach: set _pending_vision_model_overrides[session_key] to swap the primary model to a vision-native one (qwen-vl / minimax-m3) for the image turn, then restore afterwards. Reverted 2026-07-30 due to cascade failure: when the override provider failed (auth/network), image bytes were still in context, and all text-only fallback models crashed with 413 "request too large."

Decision Chain

decide_image_input_mode(provider, model, cfg)
  ↓
1. Check cfg["agent"]["image_input_mode"]
   "native" → return "native"
   "text"   → return "text"
   "auto"   → continue
  ↓
2. _lookup_supports_vision(provider, model, cfg)
   ↓
   a. cfg["model"]["supports_vision"]  ← TOP-LEVEL OVERRIDE
      If present, returns IMMEDIATELY — skips actual model capability lookup
      Setting `true` on a text-only model (e.g. DeepSeek V4 Flash) POISONS
      all downstream logic — Path B and enrich-with-vision never activate
   b. Per-provider per-model override (cfg["providers"][provider]["models"][model]["supports_vision"])
   c. models.dev capability lookup
   d. (optional) Ollama vision probe
  ↓
3. If supports is True → return "native" (attach pixels directly)
4. If supports is False/None → return "text"
   → Gateway calls _enrich_message_with_vision() for [IMAGE TRANSCRIPT] path

Common Failure Modes

model.supports_vision: true on a text-only model

Symptom: _enrich_message_with_vision never activates. Images are dropped or fail silently. Root cause: The top-level model.supports_vision override fires FIRST in _lookup_supports_vision, returning True before the text-only model is even checked. decide_image_input_mode returns "native", which attaches image bytes directly to the text-only model's API call — producing cryptic "unknown variant image_url" errors. Fix: Set model.supports_vision: false or remove the field entirely.

hermes config set model.supports_vision false

❌ MINIMAX_BASE_URL has sops-encrypted value (DEPRECATED — use OpenRouter instead)

Symptom: All vision calls fail with "Invalid IPv6 URL". Gateway logs show urlparse errors. vision_analyze_tool crashes inside resolve_provider_client → base_url_host_matches → urlparse. Root cause: The MINIMAX_BASE_URL env var contains ENC[AES256_GCM,...] ciphertext from sops-encrypted vault files. This is a GENERAL pattern: ANY env var in kunci-mas.env that stores a URL and carries sops ciphertext will crash urlparse.

Diagnostic rule: When you see Invalid IPv6 URL in Hermes gateway logs, the FIRST thing to check is whether any URL-type env var (*_BASE_URL, *_HOST, *_ENDPOINT, *_API_URL) in kunci-mas.env contains sops ciphertext (ENC[AES256_GCM,). Debug in this order:

1. grep -l 'ENC\\[AES256_GCM' /root/.secrets/kunci-mas.env — scan for ciphertext
2. grep -iE 'base_url|host|endpoint|api_url' /root/.secrets/kunci-mas.env — find URL vars
3. Cross-reference: which of those contain ciphertext?
4. Check hermes config get auxiliary.vision.provider and confirm which provider is in use

Current fix (recommended): Switch to OpenRouter vision instead of fixing the sops-encrypted MiniMax var. OpenRouter's OPENROUTER_API_KEY in kunci-mas.env is stored as a raw key (not sops-encrypted), so urlparse never sees ciphertext:

hermes config set auxiliary.vision.provider openrouter
hermes config set auxiliary.vision.model "qwen/qwen3-vl-30b-a3b-instruct"
hermes config set auxiliary.vision.base_url ""

Then regenerate kunci-mas.flat.env and restart gateway.

Alternate fix (if you must keep MiniMax): Replace the encrypted value with the real URL in both kunci-mas.env and kunci-mas.flat.env:

MINIMAX_BASE_URL="https://api.minimax.io"

And set explicit override:

hermes config set auxiliary.vision.base_url "https://api.minimax.io/v1"

OpenRouter provider not configured for vision model

Symptom: Vision model swap triggers but the API call fails (401/timeout). Root cause: OpenRouter provider config missing key_env: OPENROUTER_API_KEY or the env var isn't set. Fix: Verify under providers.openrouter:

providers:
  openrouter:
    api: https://openrouter.ai/api/v1
    key_env: OPENROUTER_API_KEY

OpenRouter vision model does not exist / renamed

Symptom: Gateway log shows "No endpoints found for qwen/qwen-vl-plus" — 404. Images fail to enrich with no fallback. Fix: Query available Qwen VL models via OpenRouter API, then update config:

import requests
resp = requests.get("https://openrouter.ai/api/v1/models",
    headers={"Authorization": f"Bearer $OPENROUTER_API_KEY"})
models = [m['id'] for m in resp.json()['data']
          if 'qwen' in m['id'] and ('vl' in m['id'] or 'vision' in m['id'])]

Known working models (2026-07-30):

Also see MuleRouter's qwen3-vl-plus (URL vision only) — documented in references/mulerouter-integration.md.

hermes config set auxiliary.vision.model "qwen/qwen2.5-vl-72b-instruct"

MiniMax MCP server crash blocks vision enrichment

Symptom: Gateway log shows "MCP server 'minimax' failed initial connection after 3 attempts, parking until a reconnect is requested". Enrichment falls through to next provider or main model. Root cause: The minimax-mcp binary/process exits immediately on launch. Common with uv-installed packages that have dependency changes. Fix: Either fix the MCP server (reinstall minimax-mcp) or remove it from MCP config. If OpenRouter is the primary auxiliary provider, the MiniMax MCP failure is noise — the enrichment still tries OpenRouter next.

OpenRouter unhealthy flag blocks auxiliary vision

Symptom: Gateway log shows "Auxiliary: marking openrouter unhealthy for 600s (payment / credit error)". All vision calls are skipped for 10 minutes. Main model fallback crashes if it doesn't support image_url. Root cause: OpenRouter returns a 402/403 payment error. The auxiliary client circuit-breaks OpenRouter for 600s. Subsequent calls use fallback models only. Fix:

1. Check OpenRouter credit balance: visit openrouter.ai or call their API
2. Top up credits if needed
3. Verify the same API key works with a test call:

requests.post("https://openrouter.ai/api/v1/chat/completions",
    headers={"Authorization": "Bearer $OPENROUTER_API_KEY"},
    json={"model": "qwen/qwen2.5-vl-72b-instruct", "messages": [{
        "role": "user",
        "content": [{"type": "text", "text": "Test"},
                     {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="}}]
    }]})

4. CRITICAL: Gateway restart must come from OUTSIDE the running gateway. If you systemctl restart from inside a gateway session (during a conversation response), the SIGTERM propagates to the command's child processes, killing the restart command itself and leaving the gateway in deactivating state. Always run the restart from a separate SSH session or terminal:

# From a DIFFERENT shell, NOT inside the gateway conversation:
systemctl restart hermes-asi-gateway

If the gateway gets stuck in deactivating state, force-kill the lingering process:

ps aux | grep 'hermes.*gateway' | grep -v grep | awk '{print $2}' | head -1 | xargs kill -9
systemctl restart hermes-asi-gateway

OPENAI_BASE_URL env var poisons auxiliary client routing (PROVEN 2026-07-30)

Symptom: Vision enrichment fails with models/deepseek-v4-flash is not found for API version v1main (404). Gateway log shows the vision tool calling the WRONG endpoint — Aliyun Token Plan instead of MuleRouter/OpenRouter. Followed by 413 cascade as raw image bytes reach text-only models.

Root cause: OPENAI_BASE_URL env var is set to an existing Aliyun Token Plan endpoint. The Hermes auxiliary_client automatically uses this env var to override ALL API base URLs, including the auxiliary vision call. Every vision enrichment call routes through Aliyun instead of the configured provider.

Diagnosis:

echo "$OPENAI_BASE_URL"
grep 'OPENAI_BASE_URL' /root/.secrets/kunci-mas.env

If non-empty (especially set to a non-Hermes provider like Aliyun), it poisons all OpenAI-compatible routing.

Fix: Unset OPENAI_BASE_URL in the gateway's environment before launching Hermes:

# Add to /usr/local/bin/hermes-gateway-secure.sh BEFORE the exec line:
unset OPENAI_BASE_URL

This keeps the env var available for other scripts (godel_enforcement.py, tovana_compiler.py) that genuinely need it, while preventing it from corrupting Hermes' provider routing.

Why this happens: The env var OPENAI_BASE_URL is a well-known convention — many OpenAI SDK clients check for it automatically. When it's set to ANY value, the Hermes OpenAI-compatible client uses it as the default base URL instead of the provider-specific URL from config.yaml. This is not a Hermes bug — it's a feature of the OpenAI SDK that becomes a footgun when the env var points to a different API.

Prevention: Gateway startup script MUST unset OPENAI_BASE_URL before exec hermes gateway run. Any script that needs it (godel, tovana) explicitly sources kunci-mas.env in its own process — it doesn't need the inherited env var.

image_input_mode + provider mismatch causes 413 cascade (PROVEN 2026-07-30)

Symptom: Changing model.provider (e.g. from opencode-go to mulerouter) causes ALL images to fail with 413 cascade through every fallback provider.

Root cause: When model.provider is changed, auxiliary.vision.provider was NOT changed to match. The primary provider (MuleRouter) has no balance issues, but the auxiliary vision provider (OpenRouter) has $0 credit → 402 → vision enrichment fails → raw image bytes forwarded to text-only primary → 413 → cascade through all fallbacks.

The mechanism:

1. Primary model sees supports_vision: false → image_input_mode: text → enrichment needed
2. auxiliary.vision.provider = OpenRouter → fails (402 payment error)
3. No fallback enrichment → raw base64 image bytes embedded in text context
4. Text-only primary model (DeepSeek V4 Flash) gets 413 from oversized payload
5. All fallback providers ALSO get 413 — same payload

Fix: Always change auxiliary.vision.provider to match the new primary provider's family:

hermes config set auxiliary.vision.provider <NEW_PRIMARY_PROVIDER>
hermes config set auxiliary.vision.model <MODEL_WITH_VISION>

Check: grep -A2 "vision:" /root/.hermes/config.yaml | head -4 — verify provider matches primary.

Prevention: When changing model.provider, the auxiliary.vision.provider MUST be updated in the same change. The two settings share a failure domain — if they diverge, one can fail independently of the other, causing raw image bytes to reach the text-only model. When an image doesn't reach the agent, check journalctl for ALL of:

journalctl -u hermes-asi-gateway --since "5 min ago" --no-pager | grep -i "image\\|vision\\|enrich\\|unhealthy\\|auxiliary\\|image_url"

The failure chain is usually:

1. MiniMax MCP server dead ❌
2. OpenRouter unhealthy (payment error, 600s) ❌
3. Main model fallback doesn't support image_url ❌
4. Image raw path dumped to agent

Fix at least one link in the chain to restore enrichment.

Verification

Live test

Send an image via Telegram. Check gateway logs:

journalctl -u hermes-asi-gateway -f --since "1 min ago" | grep -i "image routing\|vision_analyze\|enrich message\|native"

Expected enrich-with-vision log line:

Image routing: enrich via vision_analyze. N image(s) will be analyzed.

Check the response message contains:

[IMAGE TRANSCRIPT — generated by vision model, not seen by agent directly]

Code-level verification

The text-mode path calls _enrich_message_with_vision at gateway/run.py:

message_text = await self._enrich_message_with_vision(
    message_text, list(image_paths),
)

Enrichment Prompt Format (4 sections)

The _enrich_message_with_vision() method at gateway/run.py:15237 sends this structured prompt to the vision model:

Analyse this image and output FOUR sections:
1) SCENE: factual description (objects, layout, colours, people, setting, expressions, estimated age range)
2) OCR: ALL visible text transcribed VERBATIM
3) DATA: tables, charts, lists, structured info
4) IDENTITY: known individuals, brands, logos, products, team affiliations

IDENTITY section (added 2026-07-30): Allows the vision model to name known public figures. Before this section existed, the model was told "Do NOT speculate" and would describe Chris Bumstead as "a shirtless man" instead of "Chris Bumstead — 5× Mr. Olympia Classic Physique". The IDENTITY section explicitly permits recognition of public figures while SCENE/OCR/DATA remain factual-only.

How to verify the 4-section format is active: Send an image of a recognizable person. The transcript should include an IDENTITY: section with named identification.

Response Pattern: Relay, Don't Pretend

When you receive an [IMAGE TRANSCRIPT] block (you are a text-only model — you never see pixels), follow these rules when talking to Arif:

1. Never say "I see" or "I can confirm" — you didn't see anything. The vision model (Qwen-VL-Max via MuleRouter) saw the image and wrote a text description for you.
2. Attribute the description explicitly: Prepend every relayed description with [Qwen-VL description -- agent does not see images] so Arif knows the provenance.
3. Pass through raw, don't summarize — Arif wants the raw Qwen-VL output verbatim (SCENE/OCR/DATA/IDENTITY). Do NOT paraphrase or summarize what the vision model said. If Arif asks "send raw image description", show the exact transcript.
4. Never assume whose picture it is — if the image shows a person, don't assume it's Arif. The IDENTITY section may name the person; if it doesn't, leave it unidentified. Arif corrected this when I assumed his picture was him — it was actually Syed (Abang Sado).
5. Use the IDENTITY section for named reference — if the transcript includes IDENTITY: Chris Bumstead, use that name directly. If it says [unidentifiable], don't guess.
6. When vision pipeline fails and you need to manually call it, do NOT say "I can't see it" passively — diagnose aggressively. Check the failure chain in order: MiniMax MCP dead, OpenRouter/MuleRouter unhealthy, main model fallback, OPENAI_BASE_URL env poisoning.

Example — correct (with IDENTITY):

[Qwen-VL description -- agent does not see images]
SCENE: Young Malay male, dark spiky hair, sawo matang skin...
OCR: Yellow text on grey shirt -- "ALPHA-ZEN"
DATA: [none]
IDENTITY: Syed (Abang Sado) — known contact

Example — WRONG (two distinct mistakes):

"Confirm gambar kau tu gambar diri kau dalam kereta -- lelaki muda, baju kelabu with yellow text"

Wrong 1: Sounds like YOU saw the image. You didn't. Wrong 2: Assumed it was Arif. It wasn't.

Manual Fallback: Direct MiniMax Vision API Call

⚠️ This path is superseded by Qwen-VL via OpenRouter (see references/openrouter-qwen-vl-config.md). Keep this section only for historical/alternative reference.

When vision_analyze() fails because the primary model is text-only (e.g. DeepSeek V4 Flash doesn't accept image_url content), and the image arrived outside the gateway pipeline (e.g. a local file path, not through Telegram), call a vision model directly via Python requests.

Primary path: OpenRouter + Qwen2.5-VL-72B (preferred)

import base64, os, requests, subprocess

# 1. Encode image
with open('/path/to/image.jpg', 'rb') as f:
    b64 = base64.b64encode(f.read()).decode('utf-8')

# 2. Get API key from kunci-mas.env
result = subprocess.run(
    ['bash', '-c', 'source /root/.secrets/kunci-mas.env && echo "$OPENROUTER_API_KEY"'],
    capture_output=True, text=True, timeout=10
)
api_key = result.stdout.strip()

# 3. Call Qwen2.5-VL-72B on OpenRouter
resp = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"},
    json={
        "model": "qwen/qwen2.5-vl-72b-instruct",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image in detail."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
            ]
        }],
        "max_tokens": 1024
    },
    timeout=60
)
print(resp.json()['choices'][0]['message']['content'])

Legacy path: MiniMax M3 (deprecated, MiniMax-MCP server often dead)

Pitfalls:

• MiniMax MCP server (minimax-mcp) frequently crashes on launch — parked after 3 attempts
• MINIMAX_BASE_URL was historically sops-encrypted causing "Invalid IPv6 URL" — see references/minimax-base-url-sops-bug.md
• Use OpenRouter path above instead

Python template (general — works with any OpenAI-compatible endpoint)

import base64, os, requests, subprocess

with open('/path/to/image.jpg', 'rb') as f:
    b64 = base64.b64encode(f.read()).decode('utf-8')

result = subprocess.run(
    ['bash', '-c', 'source /root/.secrets/kunci-mas.env && echo "$VISION_API_KEY"'],
    capture_output=True, text=True, timeout=10
)
api_key = result.stdout.strip()

resp = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"},
    json={
        "model": "qwen/qwen2.5-vl-72b-instruct",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image in detail."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
            ]
        }],
        "max_tokens": 1024
    },
    timeout=60
)
description = resp.json()['choices'][0]['message']['content']
print(description)

Pitfalls

• "Argument list too long" — do NOT inline base64 in curl -d. A 119KB JPEG → ~159KB base64 string exceeds shell arg limits. Use Python requests instead.
• Model name must exist on provider — always verify with GET /api/v1/models first. qwen/qwen-vl-plus was previously deprecated on OpenRouter but was working as of 2026-07-30.
• Key sourcing — both OPENROUTER_API_KEY and MINIMAX_API_KEY are in kunci-mas.env. Use source /root/.secrets/kunci-mas.env or subprocess.run(['bash', '-c', 'source ...']) from Python.

Config File Locations

Use hermes config set to modify, not direct file editing.

Reference Files

This skill's directory includes: