# Forge Multimodal Router

> Federation-wide multimodal routing skill. Resolves any vision/audio/video/somatic request to its canonical model + endpoint via /root/.config/federation-models.json (SOT). USE WHEN: "which model should I use for X", "route this image/audio/video to the right engine", "unify multimodal routing across agents", "I have an image/audio/video and need to know where it goes".

- Skill: `ariffazil/forge-multimodal-router` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ariffazil/forge-multimodal-router`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ariffazil/forge-multimodal-router/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ariffazil (https://skillmd.com/u/ariffazil)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/ariffazil/forge-multimodal-router

---


# forge-multimodal-router

## Purpose

The federation supports four modalities (vision, audio, video, somatic) across 8 agents. Each modality has multiple functions (recognize, generate, edit, clone, analyze) and multiple engines (image gen, video gen, TTS, ASR, voice clone, music). Without a single routing skill, decisions are scattered across ~18 skills (`token-plan-image`, `token-plan-speech`, `token-plan-video`, `AAA-tts-engine-catalog`, `minimax-image-gen`, `creative/minimax-cli`, etc.) and 3 knowledge graphs.

This skill is the **single ingestion entry point** for any multimodal routing question. It does NOT duplicate model identity — it references the SOT.

## The authority doctrine (DO NOT VIOLATE)

| Layer | File | Owner |
|---|---|---|
| **SOT (single source of truth)** | `/root/.config/federation-models.json` | AAA/FED (Kimi-code/FI-008 writes) |
| **Reader** | `/root/AAA/scripts/fed_router.py` (`FED_SOT_PATH`) → `:7074` | FED |
| **Live health/balance** | `token_bank.db` via `:7074` (HTTP 402/200 probes) | FED runtime |
| **Tombstone** | `/root/AAA/federation/fed_signatures.yaml` — DEAD 2026-08-17 | F13 confirmed: do not resurrect |

**Rule:** Model identity lives ONLY in SOT. Skill/tool surface (Hermes config, OpenCode config, etc.) is a CONSUMER, not authority. Next model release = 1 SOT edit (RCR — Release Change Ratio).

**Picker sync rule:** AAA/FED has authority to write SOT (sqlite + json) but does NOT auto-write to picker configs (Kimi/Codex/Qwen/Claude/Go/Grok CLI). F13 holds the pen. Manual alignment audit is correct path.

## The routing decision algorithm

Given a multimodal request, ask in order:

1. **What modality?** (vision / audio / video / somatic)
2. **What direction?** (input = perception / output = generation)
3. **What function?** (recognize / generate / edit / clone / analyze / transform)
4. **What constraints?** (real-time? free quota required? Malaysian voice? identity-preserving? i-ARIF voice?)

Then resolve:

```
resolution = lookup_sot(
  modality, direction, function, constraints
)
# SOT returns: { provider_id, model_key, endpoint_url, params_override }
```

If SOT doesn't have a matching entry → STOP. Do NOT fallback to agent card or pickers. Either:
- Check `arifos-federation-provider-multimodal-discovery` skill to probe the provider
- Or forge an ephemeral skill for the new routing case

## Routing rules (read from SOT, this table is a navigation aid ONLY)

### Vision

| Request | SOT key | Engine | Notes |
|---|---|---|---|
| Image understanding (real-time, Telegram) | `dashscope/qwen-vl-max` | PRMT | Single failure domain, primary |
| Image understanding (fallback) | `dashscope/qwen3-vl-plus` | PRMT | When qwen-vl-max fails |
| OCR (multi-script) | `dashscope/qwen3-omni-flash` | OCR cascade | Tier 1 of AAA-OCR |
| OCR (Latin) | `local/tesseract-5.5` | Local | Free fallback |
| OCR (Chinese) | `local/rapidocr-3.9` | Local | Free fallback |
| Document extraction | `dashscope/qwen3-vl-ocr` | VLM specialist | See FORGE-document-intelligence |
| Image understanding (subscription-first) | `mimo/mimo-v2.5` | MiMo Token Plan | `image_url` ≤50MB URL/b64; SOT rule `vision_primary_mimo_subscribed` p12; verified live 2026-09-07 |
| Face ID (deterministic) | `local/onnx-insightface-buffalo_l-512dim` | Local | READ-ONLY, see `/root/.agents/skills/deterministic-face-id/SKILL.md` |
| Image generation (Gemini NB2 Workhorse) | `gemini/gemini-3.1-flash-image` | Nano Banana 2 | 512px–4K, 14 refs, video-in, Image Search Grounding |
| Image generation (Gemini NB2 Lite) | `gemini/gemini-3.1-flash-lite-image` | NB2 Lite | Fastest & lowest cost Gemini image gen |
| Image generation (Gemini NB Pro) | `gemini/gemini-3-pro-image` | NB Pro | Thinking mode, 4K, interleaved text+image, style refs |
| Image generation (T2I latest DashScope) | `dashscope/qwen-image-3.0-pro` | DashScope PAYG | Free quota eligible |
| Image generation (T2I fast) | `dashscope/qwen-image-3.0` | DashScope PAYG | Newest Aug 2026 |
| Image generation (T2I mass) | `dashscope/z-image-turbo` | DashScope PAYG | 8-step inference |
| Image editing (identity-preserving NB2) | `gemini/gemini-3.1-flash-image` | NB2 | 6 Iron Rules, multi-turn conversational |
| Image editing (fallback) | `dashscope/qwen-image-edit-max` | DashScope PAYG | LoRA support |
| Image generation (Malay/SEA) | `minimax/image-01` | MiniMax MCP | Default |
| Image generation (Grok Imagine T2I) | **harness** `grok-build/image_gen` | spawn `grok-multimodal.sh image` | Native Grok Build. Not FED. |
| Image editing (Grok Imagine) | **harness** `grok-build/image_edit` | spawn `grok-multimodal.sh edit REF` | Reference-first for named people |

### Audio

| Request | SOT key | Engine | Notes |
|---|---|---|---|
| Audio understanding (in-model, subscription) | `mimo/mimo-v2.5` | MiMo Token Plan | `input_audio`; MP3/WAV/FLAC/M4A/OGG; ~6.25 tok/s; verified 2026-09-07 |
| ASR (subscription-first) | `mimo/mimo-v2.5-asr` | MiMo Token Plan | wav/mp3 b64≤10MB; lang auto/zh/en (no `ms`); ~$0.074/audio-hr equiv |
| ASR (short audio <5min) | `dashscope/qwen-audio-3.0-asr-flash` | DashScope PAYG | Default |
| ASR (long offline file) | `dashscope/qwen-audio-3.0-asr-flash-filetrans` | DashScope PAYG | Meetings, calls |
| ASR (real-time stream) | `dashscope/qwen-audio-3.0-asr-flash-streaming` | DashScope PAYG | Live meetings |
| ASR (Penang-Besi dialect) | `dashscope/qwen-audio-3.0-asr-flash` + custom dict | DashScope PAYG | See AAA-asr-glm-ingest |
| ASR (default) | `local/faster-whisper-base` | Local | Free fallback |
| TTS (free window default) | `mimo/mimo-v2.5-tts` | MiMo Token Plan | 0 credits limited-time; via chat/completions `audio` field; verified live 2026-09-07 |
| TTS (default Malaysian) | `edge-tts/ms-MY-YasminNeural` | Edge (free) | No quota |
| TTS (multilingual) | `dashscope/qwen-audio-3.0-tts-flash` | DashScope PAYG | Free quota eligible |
| TTS (realtime full-duplex) | `dashscope/qwen-audio-3.0-realtime-plus` | DashScope PAYG | Aug 2026 |
| TTS (sovereign i-ARIF) | `minimax/speech-2.8-hd` | MiniMax | voice_id `i-ARIF-20260819T084602` |
| TTS (Penang-Besi dialect) | `mimo/mimo-v2.5-tts-voicedesign` | MiMo Token Plan | See nusantara-acoustic-infrastructure |
| TTS (voice clone) | `dashscope/voice-enrollment` | DashScope PAYG | + qwen-voice-design |
| DSP analysis | `local/librosa` | Local | See media/audio-analysis |
| Live translate (realtime) | `dashscope/qwen3.5-livetranslate-flash-realtime` | DashScope PAYG | Aug 2026 |

### Video

| Request | SOT key | Engine | Notes |
|---|---|---|---|
| Video generation & editing (Omni default) | `gemini/gemini-omni-flash` | Gemini Omni Flash | ⭐ DEFAULT video model: multi-turn conversational editing & omnimodal reasoning |
| Video generation (Cinematic + Native Audio) | `gemini/veo-3.1-generate-preview` | Veo 3.1 | Native dialogue/SFX/ambient, 720p/1080p/4k, first/last frame, up to 3 refs |
| Video extension (Temporal sequence) | `gemini/veo-3.1-generate-preview` | Veo 3.1 | Extend Veo video by 7s (up to 20x, max 148s, 720p) |
| Video generation (Veo Fast) | `gemini/veo-3.1-fast-generate-preview` | Veo 3.1 Fast | Fast video gen with native audio |
| Video generation (Veo Lite) | `gemini/veo-3.1-lite-generate-preview` | Veo 3.1 Lite | Lightweight video gen (720p/1080p) |
| Video generation (T2V DashScope newest) | `dashscope/wan3.0-video` | DashScope PAYG | Aug 2026 |
| Video generation (T2V standard) | `dashscope/wan2.7-t2v-2026-06-12` | DashScope PAYG | Jul 2026 |
| Video generation (I2V) | `dashscope/wan2.7-i2v-2026-04-25` | DashScope PAYG | Preserve subject/style |
| Video generation (R2V) | `dashscope/wan2.7-r2v-2026-06-12` | DashScope PAYG | Up to 9 refs |
| Video generation (R2V alternative) | `dashscope/happyhorse-1.1-r2v` | DashScope PAYG | Realistic dynamic |
| Video generation (Grok Imagine I2V) | **harness** `grok-build/image_to_video` | spawn `grok-multimodal.sh video REF` | 6s/10s · 720p. Native Grok Build. |
| Video editing (DashScope) | `dashscope/wan2.7-videoedit` | DashScope PAYG | Local/global edits |
| Video understanding (subscription) | `mimo/mimo-v2.5` | MiMo Token Plan | `video_url` + fps 0.1–10 + media_resolution; ≤300MB URL / 50MB b64; MP4/MOV/AVI/WMV; only subscription video-understanding rung; verified 2026-09-07 |
| Last-frame analysis | `dashscope/qwen-vl-max` (re-use) | PRMT | No temporal context |
| Video INPUT (continuous) | `gemini/gemini-omni-flash` | Gemini Omni Flash | Native video input reasoning via Interactions API |
| A-V cross-modal fusion | `gemini/gemini-omni-flash` | Gemini Omni Flash | Text + Image + Audio + Video unified input |

### Somatic (Music Intelligence)

| Request | SOT key | Engine | Notes |
|---|---|---|---|
| Music generation (governed) | `minimax/music-2.6` | MiniMax T2A Music | Per AAA-somatic-emd-pipeline |
| Music generation (newest) | `minimax/music-3.0` | MiniMax T2A Music | |
| Music scoring (somatic) | `federation/somatic-scorer` | WELL + scoring | Per AAA-somatic-music-doctrine |
| Songwriting + Suno | `suno/suno-v3` | Suno | Creative lane |
| DSP feature analysis | `local/librosa-scipy` | Local | See media/audio-feature-analysis |

## Constraints & override rules

### Free quota preference
- If SOT model is in Singapore region AND has International scope AND has blue quota bar in console → prefer it over PAYG
- Validate via `arifos-federation-provider-multimodal-discovery` skill before assuming free quota
- Enable **Free Quota Only** mode per model in console (default OFF for verified users)

### i-ARIF voice identity
- F13 SOVEREIGN-gated: voice identity is biometric-class
- Use `AAA-voice-cloning-mimo-minimax` for any i-ARIF request
- All i-ARIF calls seal to VAULT999 with explicit voice_id annotation

### Identity preservation (face/body)
- F13, 2026-08-12: "Hang jangan ubah muka manusia"
- Real photo edits MUST preserve face, body, skin tone 100%
- Use `media/aaa-image-editing` (6 Iron Rules) + ensemble NB2 + NB-Pro for identity-critical work

### Privacy / Sovereignty
- Voice biometric = F1 custody (vault-class)
- Face ID = local-only (no cloud round-trip)
- All auth keys via `~/.secrets/kunci-root.env` (5-R Protocol)

## How to invoke this skill

When an agent receives a multimodal request and needs to route:

1. **Classify** the modality + direction + function + constraints
2. **Look up** the SOT key (use this skill's routing table as index, then verify against `/root/.config/federation-models.json`)
3. **Verify** the SOT entry is current (F11: log the lookup with SHA256 of SOT file)
4. **Resolve** to `provider_id + model_key + endpoint_url`
5. **Call** the model via the agent's MCP surface (e.g., `mcp__aforge__forge_browser_*` for browser, or direct API call for non-MCP)
6. **If the route is Grok Imagine**: spawn `/root/.grok/bin/grok-multimodal.sh`. Those tools are harness-native to FI-007, not Hermes/OpenCode.
7. **Log** the routing decision to VAULT999 (F11 AUDIT)

### Harness-native Imagine (not a FED model)

Telephone: `/root/AAA/docs/GROK_IMAGINE.md`. Wrapper: `grok-multimodal`. **Do not** add grok to LiteLLM.

## Cross-cutting doctrine (apply across modalities)

| Concern | Skill | Authority |
|---|---|---|
| Provider capability probing | `arifos-federation-provider-multimodal-discovery` | Verifies before claiming |
| Visual structural QA | `forge-vss-verifier-suite` | Post-generation |
| Multimodal reasoning | `AGI-multimodal-bridge` | Cross-modal fusion |
| Audio quantum doctrine | `AGI-audio-quantum-cognition` | Audio substrate |
| Visual substrate | `delta-omega-psi-multimodal-cognition` | Δ·Ω·Ψ enforcement |
| Visual governance | `FORGE-visual-qa-w3` | W³ tri-witness |
| TTS engine registry | `AAA-tts-engine-catalog` | TTS routing detail |
| Voice cloning | `AAA-voice-cloning-mimo-minimax` + `AAA-voice-cloning-qwen-cloud` | Voice identity |
| Somatic music | `AAA-somatic-music-doctrine` + `AAA-somatic-emd-pipeline` | Music constitution |
| Document intelligence | `FORGE-document-intelligence` | Doc VLM |
| OCR cascade | `AAA-OCR-optical-compression` | OCR Tier 1-3 |
| Face ID | `deterministic-face-id` | READ-ONLY, peer skill at `/root/.agents/skills/deterministic-face-id/` |
| Cross-region config | `devops/provider-routing-zen` | Operational |

## Knowledge graphs (canonical unified maps)

| Modality | KG file | Lines | Forged |
|---|---|---|---|
| Audio | `/root/AAA/knowledge-graph/audio-intelligence-map.md` | 338 | 2026-08-13 |
| Visual | `/root/AAA/knowledge-graph/visual-intelligence-map.md` | 911 | 2026-08-18 |
| Video | `/root/AAA/knowledge-graph/video-intelligence-map.md` | 529 | 2026-08-18 |
| Somatic | **MISSING** — use `AAA-somatic-music-doctrine` SKILL.md | — | — |

## Floors

- **F2 TRUTH:** Every routing decision cites the SOT entry used (provider_id + model_key + SHA256 of SOT file at lookup time).
- **F4 CLARITY:** This skill is a navigation index, not a duplicate of SOT. Single source of truth = SOT.
- **F8 GENIUS:** Cross-modal requests route through η algorithm (4 questions) before falling back to per-modality skills.
- **F11 AUDIT:** Every routing decision logged to VAULT999 receipts (provider + model + token estimate + SOT version).
- **F1 AMANAH:** Free quota preference + BlueQuotaBar check before PAYG. No surprise charges.

## Annual verification

This skill MUST be re-validated:
- On any SOT change to `/root/.config/federation-models.json` (RCR enforcement)
- On any new model release touching vision/audio/video/somatic
- Quarterly (next due: 2026-11-20)

If the routing table here drifts from SOT, the table is wrong (SOT wins). Update this skill to match SOT, never the reverse.

## Failure modes

| Failure | Action |
|---|---|
| SOT missing entry for modality+function | Use `arifos-federation-provider-multimodal-discovery` to probe provider; if real, propose SOT edit (F13 stance: 1 SOT edit per model release) |
| Provider returns 402 (insufficient balance) | Trigger `devops/qwen-provider-operations` for diagnosis |
| Provider returns 401 (auth fail) | Verify key in `~/.secrets/kunci-root.env`; rotate if needed |
| Hermes config drift from SOT | Hermes config is consumer; SOT is authority. Patch Hermes config to match SOT, NOT vice versa. |
| Cross-agent picker drift | F13 holds the pen. AAA writes SOT only, does NOT auto-sync to pickers. Manual alignment audit. |

## Audit trail

- **2026-08-25** — FI-007: Imagine spawn path for all AAA agents (`grok-multimodal.sh` + GROK_IMAGINE.md). Harness-native, not FED.
- **2026-08-20** — forged by FI-003 Qwen Code
- Driver: Arif F13 directive "now make sure my hermes agent telegram ASI_arifos_bot know how to use this"
- Builds on: 32 multimodal models wired into `/root/.hermes/config.yaml` `dashscope-payg` provider + `tts/stt.qwen-audio-payg` entries (2026-08-20)
- Builds on: 3 canonical knowledge graphs (audio + visual + video) — `/root/AAA/knowledge-graph/`
- Constraints: F2 SOT authority, F11 AUDIT, F13 picker sovereignty, RCR 1-edit-per-release
- Receipt: to be appended to `/root/forge_work/qwen-sessions/sessions.jsonl` + named receipt