Model Routing Selector
Overview
The horizontal workflow graph (Chapters 5-6) decomposes the agent into
specialized nodes: a query analyst, a retrieval strategist, a synthesis node, a
validation node. Most teams start with the same frontier model powering every
node. It works, and it is wildly expensive.
Selective intelligence matches model capability to task complexity. Running a
3B SLM can be 10-30x cheaper per token than its 405B sibling; at thousands or
millions of invocations per day, that difference decides whether the project
survives its first budget review. The chapter's key insight is that most nodes
do not need frontier reasoning — a fine-tuned 3B model classifies alerts, a
3.8B Triplex model beats GPT-4o at knowledge-graph construction, and only the
open-ended synthesis and causal-reasoning nodes (10-20% of invocations)
reliably benefit from a frontier model.
This skill re-derives the book's DevOps assignment (Example 8-13) from a node's
required_quality bar and a per-model cost/capability catalog, so you can see
why each node gets the model it gets rather than accepting an opaque config.
When to Use
- A pipeline runs every node on one frontier model and cost is unsustainable.
- You are adding a node and need to know the cheapest model that clears its bar.
- A node's per-query difficulty varies and you are deciding static vs cascade.
- You have production traffic and want to justify a learned router.
Phrases that should invoke this skill: "which model for this node", "selective
intelligence", "cheapest model that meets the bar", "static vs cascade routing",
"cut inference cost", "RouteLLM", "FrugalGPT cascade".
When NOT to Use
- Single-model systems. With one node type and one model there is nothing
to route.
- Choosing the graph data model (property graph vs RDF vs hypergraph) — that
is
graph-model-selector (Ch3).
- Measuring a routing policy after deployment — that is
cost-performance-scorer (Ch8). This skill decides routing before you run;
that skill scores what actually happened.
- KV-cache / latency budgeting — that is
kv-cache-latency-budgeter (Ch8).
Routing lowers cost per weight; it does not lower the concurrency ceiling.
Process
| Step |
Input |
Action |
Output |
Verification |
| 1 |
Node name |
lib.NODES[name] |
Node with task_class + required_quality |
Node exists; bar in [0,1] |
| 2 |
Node |
lib.cheapest_meeting_bar(name) |
Cheapest model whose effective quality clears the bar |
effective_quality >= required_quality, or a cascade recommendation |
| 3 |
Node + confidence |
lib.cascade_route(name, confidence) |
Served tier (cheap/mid/frontier) |
High confidence -> cheap SLM; low -> frontier |
| 4 |
Node + cost threshold |
lib.learned_route(name, t) |
RouteLLM-style model id |
router-mf-<t>; lower t -> weak model |
| 5 |
Escalation rate |
lib.pipeline_cost(rate) |
Blended cost vs frontier-everywhere |
reduction_pct >= 65% equal-weight (book: ~80% token-weighted) |
| 6 |
Selected model |
(your runtime) invoke via vLLM multi-LoRA / API |
Model response |
If quality below bar in production, re-evaluate with cost-performance-scorer |
Rationalizations
| Agent rationalization |
Documented rebuttal |
| "Context windows are cheap now — just use the frontier model everywhere." |
The book's economics are per-token, not per-window. A 3B SLM is 10-30x cheaper per token; at production volume the frontier-everywhere invoice is what shelves the project (Ch8 opening). |
| "A small model can't be trusted on a high-stakes node." |
The AlertClassifier needs 0.99 accuracy AND runs on a 3B model — because a fine-tuned specialist reaches the bar on a narrow task. Capability on the task, not raw size, is the criterion (Example 8-4, Triplex, Kakao 63% preference). |
| "Cascades add latency, skip them." |
For nodes where difficulty varies, a cascade serves the easy majority cheaply and escalates the hard minority. Static routing to the frontier pays frontier cost on every query, including the 80% that a mid-tier model handles (Ch8 threshold-based cascading). |
| "Learned routing is the sophisticated choice, always use it." |
RouteLLM/MixLLM need production traffic to train and engineering to maintain. The book: static routing or threshold cascading covers 80% of the benefit at 20% of the complexity. Start static. |
| "Route CausalAttributionNode straight to the frontier — it's reasoning." |
It is open-ended, so a small model can't handle every case, but it handles the routine ones. The cascade (8B -> sonnet, escalate below 0.7) is why blended cost lands at ~1/5th, not 1x (Example 8-13). |
Red Flags
- Every node routes to the frontier model. Either the bars are set too high
or the specialization mechanism is disabled. Re-check
effective_quality.
- A 3B model is selected for open-ended synthesis. The specialist ceiling
only applies to
classification/extraction task classes; synthesis must
fall through to a capable model.
- Blended reduction below 70%. The pipeline is mis-tiered — probably a
routine node is still on the frontier model.
- Cascade thresholds set so high nothing is served cheap. You lose the
savings; calibrate against a per-node eval set (cost-performance-scorer).
- CLI
--help exits non-zero. SKILL.md / CLI mismatch; the multi-harness
invariant is broken.
Non-Negotiable Verification
Before shipping a routing policy built on this skill:
Run the benchmark battery.
python cli.py benchmark
Confirms the router re-derives the book's DEVOPS_MODEL_CONFIG for all five
nodes and that blended reduction is >= 70%.
Inspect the routing decision for a high-stakes node.
python cli.py route AlertClassifier
python cli.py route CausalAttributionNode
Confirm AlertClassifier picks the 3B SLM (specialist meets 0.99) and
CausalAttributionNode is returned as a cascade, not raw frontier.
Confirm the blended-cost math.
python cli.py pipeline --escalation-rate 0.3
Equal-weight nodes land near 70%; the book's ~80% figure assumes token
weighting where the frontier synthesis node dominates the baseline. Either
way the reduction is the difference between a viable production system and a
budget overrun (Ch8).
Domain test in the notebook. Run notebooks/ch8-optimization.ipynb;
confirm the selective-intelligence section drives per-node model assignment
and the blended cost is computed against moto-mocked CloudWatch cost signals.
Security Posture
- No model invocation here.
lib.py makes no network calls; it returns a
model identifier and rationale. The caller invokes the model.
- Cost figures are advisory. The cost multipliers are book figures for
relative reasoning; real per-token pricing changes. Recompute against live
pricing before committing a budget.
- Routing-as-attack-surface. If a learned router ingests query text, treat
that text as untrusted (an adversarial query could bias routing toward an
expensive or a too-weak model). Sanitize before the router sees it — the
Retrieved-Content Adversarial-Input discipline applies at the router seam.
Composition
- Composes with
cost-performance-scorer (Ch8): this skill decides routing
a priori; that skill measures cost-per-successful-completion a posteriori and
feeds the calibration back into the thresholds here.
- Composes with
kv-cache-latency-budgeter (Ch8): routing lowers cost per
weight; the budgeter checks the resulting fleet against the concurrency and
latency ceiling.
- Feeds the vLLM multi-LoRA serving pattern (Ch8 Example 8-12): the selected
specialist adapters share one base model on one GPU.
Source Attribution
Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien),
Chapter 8 — Optimization, "Selective Intelligence" / "Routing Strategies".
Key references named in the chapter:
- NVIDIA Research SLM-for-agentic-AI position paper (front-door router)
- FrugalGPT (threshold-based cascade)
- RouteLLM (>2x cost reduction, 95% GPT-4 quality on MT-Bench)
- MixLLM (contextual bandit; 97% GPT-4 quality at 24% cost)
- Triplex 3.8B (beats GPT-4o at KG construction)
- Kakao AI Shopping Mate (format adherence 0.655 -> 0.987, accuracy 0.578 -> 0.890,
smaller fine-tuned models preferred over GPT-4o in 63% of 2,100 turns)
- Example 8-13 DEVOPS_MODEL_CONFIG (the derivation target for this skill)
1---2name: model-routing-selector3description: Match model capability to task complexity across a horizontal workflow graph. Given a node, pick the cheapest model that meets its quality bar, using one of three routing strategies: static routing by node type, threshold-based cascading (FrugalGPT), or learned routing (RouteLLM / MixLLM). Re-derives the book's DevOps DEVOPS_MODEL_CONFIG from first principles and reports the blended cost reduction (~80%). Use when every node of an agentic pipeline runs on the same frontier model and the invoice is unsustainable. NOT for single-model systems, NOT for choosing WHICH graph model to use (that is graph-model-selector), NOT for measuring a policy after the fact (that is cost-performance-scorer).4---56# Model Routing Selector78## Overview910The horizontal workflow graph (Chapters 5-6) decomposes the agent into11specialized nodes: a query analyst, a retrieval strategist, a synthesis node, a12validation node. Most teams start with the same frontier model powering every13node. It works, and it is wildly expensive.1415Selective intelligence matches model capability to task complexity. Running a163B SLM can be 10-30x cheaper per token than its 405B sibling; at thousands or17millions of invocations per day, that difference decides whether the project18survives its first budget review. The chapter's key insight is that most nodes19do not need frontier reasoning — a fine-tuned 3B model classifies alerts, a203.8B Triplex model beats GPT-4o at knowledge-graph construction, and only the21open-ended synthesis and causal-reasoning nodes (10-20% of invocations)22reliably benefit from a frontier model.2324This skill re-derives the book's DevOps assignment (Example 8-13) from a node's25`required_quality` bar and a per-model cost/capability catalog, so you can see26*why* each node gets the model it gets rather than accepting an opaque config.2728## When to Use2930- A pipeline runs every node on one frontier model and cost is unsustainable.31- You are adding a node and need to know the cheapest model that clears its bar.32- A node's per-query difficulty varies and you are deciding static vs cascade.33- You have production traffic and want to justify a learned router.3435Phrases that should invoke this skill: "which model for this node", "selective36intelligence", "cheapest model that meets the bar", "static vs cascade routing",37"cut inference cost", "RouteLLM", "FrugalGPT cascade".3839## When NOT to Use4041- **Single-model systems.** With one node type and one model there is nothing42 to route.43- **Choosing the graph data model** (property graph vs RDF vs hypergraph) — that44 is `graph-model-selector` (Ch3).45- **Measuring a routing policy after deployment** — that is46 `cost-performance-scorer` (Ch8). This skill decides routing *before* you run;47 that skill scores what actually happened.48- **KV-cache / latency budgeting** — that is `kv-cache-latency-budgeter` (Ch8).49 Routing lowers cost per weight; it does not lower the concurrency ceiling.5051## Process5253| Step | Input | Action | Output | Verification |54|------|-------|--------|--------|--------------|55| 1 | Node name | `lib.NODES[name]` | Node with task_class + required_quality | Node exists; bar in [0,1] |56| 2 | Node | `lib.cheapest_meeting_bar(name)` | Cheapest model whose effective quality clears the bar | `effective_quality >= required_quality`, or a cascade recommendation |57| 3 | Node + confidence | `lib.cascade_route(name, confidence)` | Served tier (cheap/mid/frontier) | High confidence -> cheap SLM; low -> frontier |58| 4 | Node + cost threshold | `lib.learned_route(name, t)` | RouteLLM-style model id | `router-mf-<t>`; lower t -> weak model |59| 5 | Escalation rate | `lib.pipeline_cost(rate)` | Blended cost vs frontier-everywhere | `reduction_pct` >= 65% equal-weight (book: ~80% token-weighted) |60| 6 | Selected model | (your runtime) invoke via vLLM multi-LoRA / API | Model response | If quality below bar in production, re-evaluate with cost-performance-scorer |6162## Rationalizations6364| Agent rationalization | Documented rebuttal |65|-----------------------|---------------------|66| "Context windows are cheap now — just use the frontier model everywhere." | The book's economics are per-token, not per-window. A 3B SLM is 10-30x cheaper per token; at production volume the frontier-everywhere invoice is what shelves the project (Ch8 opening). |67| "A small model can't be trusted on a high-stakes node." | The AlertClassifier needs 0.99 accuracy AND runs on a 3B model — because a *fine-tuned specialist* reaches the bar on a narrow task. Capability on the task, not raw size, is the criterion (Example 8-4, Triplex, Kakao 63% preference). |68| "Cascades add latency, skip them." | For nodes where difficulty varies, a cascade serves the easy majority cheaply and escalates the hard minority. Static routing to the frontier pays frontier cost on every query, including the 80% that a mid-tier model handles (Ch8 threshold-based cascading). |69| "Learned routing is the sophisticated choice, always use it." | RouteLLM/MixLLM need production traffic to train and engineering to maintain. The book: static routing or threshold cascading covers 80% of the benefit at 20% of the complexity. Start static. |70| "Route CausalAttributionNode straight to the frontier — it's reasoning." | It is open-ended, so a small model can't handle *every* case, but it handles the routine ones. The cascade (8B -> sonnet, escalate below 0.7) is why blended cost lands at ~1/5th, not 1x (Example 8-13). |7172## Red Flags7374- **Every node routes to the frontier model.** Either the bars are set too high75 or the specialization mechanism is disabled. Re-check `effective_quality`.76- **A 3B model is selected for open-ended synthesis.** The specialist ceiling77 only applies to `classification`/`extraction` task classes; synthesis must78 fall through to a capable model.79- **Blended reduction below 70%.** The pipeline is mis-tiered — probably a80 routine node is still on the frontier model.81- **Cascade thresholds set so high nothing is served cheap.** You lose the82 savings; calibrate against a per-node eval set (cost-performance-scorer).83- **CLI `--help` exits non-zero.** SKILL.md / CLI mismatch; the multi-harness84 invariant is broken.8586## Non-Negotiable Verification8788Before shipping a routing policy built on this skill:89901. **Run the benchmark battery.**91 ```92 python cli.py benchmark93 ```94 Confirms the router re-derives the book's DEVOPS_MODEL_CONFIG for all five95 nodes and that blended reduction is >= 70%.96972. **Inspect the routing decision for a high-stakes node.**98 ```99 python cli.py route AlertClassifier100 python cli.py route CausalAttributionNode101 ```102 Confirm AlertClassifier picks the 3B SLM (specialist meets 0.99) and103 CausalAttributionNode is returned as a *cascade*, not raw frontier.1041053. **Confirm the blended-cost math.**106 ```107 python cli.py pipeline --escalation-rate 0.3108 ```109 Equal-weight nodes land near 70%; the book's ~80% figure assumes token110 weighting where the frontier synthesis node dominates the baseline. Either111 way the reduction is the difference between a viable production system and a112 budget overrun (Ch8).1131144. **Domain test in the notebook.** Run `notebooks/ch8-optimization.ipynb`;115 confirm the selective-intelligence section drives per-node model assignment116 and the blended cost is computed against `moto`-mocked CloudWatch cost signals.117118## Security Posture119120- **No model invocation here.** `lib.py` makes no network calls; it returns a121 model *identifier* and rationale. The caller invokes the model.122- **Cost figures are advisory.** The cost multipliers are book figures for123 relative reasoning; real per-token pricing changes. Recompute against live124 pricing before committing a budget.125- **Routing-as-attack-surface.** If a learned router ingests query text, treat126 that text as untrusted (an adversarial query could bias routing toward an127 expensive or a too-weak model). Sanitize before the router sees it — the128 Retrieved-Content Adversarial-Input discipline applies at the router seam.129130## Composition131132- **Composes with** `cost-performance-scorer` (Ch8): this skill decides routing133 a priori; that skill measures cost-per-successful-completion a posteriori and134 feeds the calibration back into the thresholds here.135- **Composes with** `kv-cache-latency-budgeter` (Ch8): routing lowers cost per136 weight; the budgeter checks the resulting fleet against the concurrency and137 latency ceiling.138- **Feeds** the vLLM multi-LoRA serving pattern (Ch8 Example 8-12): the selected139 specialist adapters share one base model on one GPU.140141## Source Attribution142143Distilled from *Agentic GraphRAG* (O'Reilly, by Anthony Alcaraz and Sam Julien),144Chapter 8 — Optimization, "Selective Intelligence" / "Routing Strategies".145Key references named in the chapter:146147- NVIDIA Research SLM-for-agentic-AI position paper (front-door router)148- FrugalGPT (threshold-based cascade)149- RouteLLM (>2x cost reduction, 95% GPT-4 quality on MT-Bench)150- MixLLM (contextual bandit; 97% GPT-4 quality at 24% cost)151- Triplex 3.8B (beats GPT-4o at KG construction)152- Kakao AI Shopping Mate (format adherence 0.655 -> 0.987, accuracy 0.578 -> 0.890,153 smaller fine-tuned models preferred over GPT-4o in 63% of 2,100 turns)154- Example 8-13 DEVOPS_MODEL_CONFIG (the derivation target for this skill)
AnthonyAlcaraz