LLM Routing Skill

LLM Routing Skill

Route tasks and agent sub-steps to the right model — the one that sits on the cost–quality Pareto frontier for the task's required capability level, under your cost tier, latency, and governance constraints. The routing intelligence is a living graph of model capability × cost × latency (RDF-Turtle + JSON mirror in references/), rebuilt from live pricing feeds whenever prices change or capability profiles are refined via feedback.

This is the routing contract. Read the whole file before routing; re-read the relevant section before building any routing artifact (Anti-Drift Protocol).

1. Why this exists

A static "always use the biggest model" policy wastes money. A pure cheapest-first policy risks quality failures. The sweet spot is the cost–quality Pareto frontier:

1. Map the task (or agent sub-step) to a required capability level.
2. Know which models deliver acceptable quality for that capability, at what price and latency — from the routing graph.
3. Route simple/repetitive work to efficient models and escalate only when needed (advisor pattern).

The same intelligence powers OpenRouter's Auto Router (task classification + community share-of-spend + your cost_tier), Snowflake Cortex's dynamic model routing (approved models + trade-off policies + classifier + advisor), and RouteLLM-style cascades. This skill makes that intelligence yours: transparent, queryable, and editable — the graph, the knobs, and the feedback loop are all first-class artifacts.

2. Architecture

                    ┌──────────────────────────────────────────────┐
                    │            LLM ROUTING GRAPH                 │
                    │  references/routing-graph.ttl (+ .json)      │
                    │  model capability × cost × latency           │
                    │  per task type: Pareto frontier + escalation │
                    └──────────────────────────────────────────────┘
                       ▲ rebuild                    │ query
        ┌──────────────┴─────────────┐              ▼
  ┌─────┴──────┐  ┌──────────────────┴───┐  ┌───────────────┐
  │ PRICES     │  │ PROFILES (seeds)      │  │ ROUTER        │
  │ live       │  │ capability-profiles   │  │ classify task │
  │ llm-prices │  │ .json — edit me       │  │ pick policy   │
  │ .com feeds │  │ task-types.json       │  │ query graph   │
  └────────────┘  └──────────────────────┘  │ escalate      │
                                            └───────────────┘

Ontology

The graph uses the llmr: namespace https://www.openlinksw.com/ontology/llm-routing# (documented pattern — model entities under llmrm:, task types under llmrt:, vendors under llmrv:, capability levels under llmrl:), plus schema.org for model/vendor typing, ordered collections (schema:ItemList/ListItem/position for escalation ladders), and attribute pairs (schema:additionalProperty + schema:PropertyValue for capability profiles). Sets are repeated triples, never RDF lists (llmr:paretoFrontierModel, llmr:dominantDimension). intel: (Intelligence Allocation Ontology, https://linkeddata.uriburner.com/DAV/demos/daas/ontology/intelligence-allocation#) is the linked concept for tier/execution-surface semantics. New terms are registered in agent-rdf-memory/entities/ontology-terms.ttl per the cross-document-local-term-reuse rule.

3. Build & Refresh (dynamic pricing)

Prices come from live feeds — never hard-code them:

# 1. fetch the latest prices (cached to scripts/.cache/)
python3 scripts/fetch_prices.py

# 2. rebuild the graph (uses .cache, or fetches live, or local llm-prices checkout)
python3 scripts/build_routing_graph.py

# 3. GATE — must pass before using the graph
python3 scripts/validate_graph.py

Options:

• --prices /path/to/current-v1.json — use a specific feed snapshot.
• --offline — build from the local llm-prices git checkout at ~/Documents/Management/Development/llm-prices/data/ (no network).
• --out-dir /tmp/x — write elsewhere.

Refresh cadence: re-fetch + rebuild whenever a model's price changes or a new model appears (llm-prices updates weekly-ish), and after every feedback round that edits capability profiles. validate_graph.py fails if prices are older than --max-age-days (default 30).

4. Routing Workflow

Step 1 — Classify the task into a task type

Pick the best match from the ~30 task types in references/task-types.json (list: python3 scripts/route.py --list-tasks). For an agent sub-step, classify the sub-step, not the whole goal. When a task straddles two types, use the higher required capability level.

Step 2 — Determine the required capability level

Each task type declares required_level (L1–L5):

Step 3 — Choose the cost tier and policy

Cost tiers (mirroring OpenRouter's cost_tier), based on output price per million tokens:

Policies:

Step 4 — Query the graph and decide

CLI (fast, no dependencies):

python3 scripts/route.py code-generation medium
python3 scripts/route.py logical-reasoning max --policy quality-first
python3 scripts/route.py summarization low --policy cost-first --latency medium --vendor google
python3 scripts/route.py tool-calling high --json

SPARQL over the TTL (full expressiveness — e.g., governance or cross-vendor queries); see §5 for templates. Decision rule of thumb: start at the cheapest frontier model at the required level; escalate only when output quality fails the task's verification gate.

Step 5 — Escalate (advisor pattern)

Every task in the graph carries an escalationLadder (schema:ItemList of ListItem → model, ordered by position): cheapest adequate model → cheapest at the next capability level → best model in the graph. Execute the task on the first rung; verify the output; on failure, move up a rung. This mirrors Snowflake Cortex's "try a smaller model first, escalate if needed".

Step 6 — Record feedback

After execution, score output quality (1–5) and record it in references/feedback-log.ttl (template in §7). Feedback that contradicts the graph's seed capability scores feeds the profile-refinement loop (§7), keeping the graph fresh as models improve and prices drop — the "living graph" property.

5. SPARQL Query Templates

Run against references/routing-graph.ttl (rdflib, or load into a Virtuoso quad store).

T1 — Models meeting a capability level for a task, cheapest first:

PREFIX llmr: <https://www.openlinksw.com/ontology/llm-routing#>
PREFIX llmrt: <https://www.openlinksw.com/ontology/llm-routing/tasks/>
SELECT ?model ?price WHERE {
  ?model a llmr:Model ; llmr:outputPricePerMTok ?price .
  llmrt:code-generation llmr:paretoFrontierModel ?model .
}
ORDER BY ?price

T2 — Pareto frontier for a task:

PREFIX llmr: <https://www.openlinksw.com/ontology/llm-routing#>
PREFIX llmrt: <https://www.openlinksw.com/ontology/llm-routing/tasks/>
SELECT ?model ?price ?latency WHERE {
  llmrt:logical-reasoning llmr:paretoFrontierModel ?model .
  ?model llmr:outputPricePerMTok ?price ; llmr:latencyClass ?latency .
}
ORDER BY ?price

T3 — Escalation ladder for a task (ordered via schema:ItemList):

PREFIX llmr: <https://www.openlinksw.com/ontology/llm-routing#>
PREFIX llmrt: <https://www.openlinksw.com/ontology/llm-routing/tasks/>
PREFIX schema: <http://schema.org/>
SELECT ?model WHERE {
  llmrt:code-generation schema:itemListElement ?li .
  ?li schema:position ?pos ; schema:item ?model .
}
ORDER BY ?pos

T4 — Models within a cost tier with cached-input pricing (cost-sensitive retrieval/RAG workloads):

PREFIX llmr: <https://www.openlinksw.com/ontology/llm-routing#>
SELECT ?model ?output ?cached WHERE {
  ?model a llmr:Model ;
         llmr:costTier "low"@en ;
         llmr:outputPricePerMTok ?output ;
         llmr:cachedInputPricePerMTok ?cached .
}
ORDER BY ?output

T5 — Capability profile of a model (all dimensions, schema:PropertyValue):

PREFIX llmr: <https://www.openlinksw.com/ontology/llm-routing#>
PREFIX schema: <http://schema.org/>
SELECT ?dimension ?score WHERE {
  ?model a llmr:Model ;
         schema:additionalProperty ?pv .
  ?pv schema:name ?dimension ; schema:value ?score .
  FILTER(CONTAINS(STR(?model), "gpt-5"))
}

Encoding note. Sets are serialized as repeated triples (llmr:paretoFrontierModel, llmr:dominantDimension) — query them directly, no list traversal needed. Ordered sequences (escalationLadder) use schema:ItemList/schema:ListItem/schema:position. Capability attributes use schema:additionalProperty + schema:PropertyValue. No blank nodes (preferences.ttl Step 37): every schema:ListItem and schema:PropertyValue reification node carries a named hash IRI (e.g. <.../tasks/code-generation#rung-1>, <.../models/gpt-5.6-luna#capability-reasoning>), emitted with full <IRI> syntax because # is not legal in a prefixed-name local part. Ontology terms are defined, not just used: the authoritative TBox is references/llm-routing-ontology.ttl (every llmr: class/property with rdfs:label, rdfs:comment, rdfs:domain/range, rdfs:isDefinedBy, and verified external cross-references per the ontology-cross-reference gate), embedded verbatim into routing-graph.ttl at build time; the validator fails if any used llmr: term is undeclared or any blank node appears. All templates above were executed against the shipped graph and verified to return rows.

6. Governance

You keep the knobs — routing intelligence is advisory, governance is yours:

• Approved models / blocked models / residency constraints: edit the governance block of references/capability-profiles.json; the builder projects them into the graph (llmr:approvesModel, llmr:blocksModel, llmr:residencyConstraint). Restrict route.py with --vendor or filter in SPARQL.
• Cost caps: the cost tier IS the cap; choose per workload, not globally.
• PII / residency: add residency constraints per region; route.py does not know your data-classification policy — apply it before calling the model.
• Audit: routing decisions are reproducible — graph snapshot + task type + tier + policy fully determine the recommendation. Record decisions in the feedback log for auditability.

7. Feedback Loop (keeps the graph alive)

Two feedback paths — manual (explicit FeedbackRecords) and automatic (session routing traces harvested into seed refinements). Both are PRIVATE: recorded locally, never uploaded to public surfaces.

7a. Session traces (automatic — preferred)

Routing sessions record an outcome-level llmr:RoutingTrace per routed execution, in line with the session-trace guidelines in agent-rdf-memory/preferences.ttl (secret redaction, opal:ChatSession linkage via prov:wasInformedBy, intent-to-outcome traceability):

python3 scripts/record_trace.py code-generation deepseek-v4-flash \
    --tier medium --policy balanced --score 4 --latency low --cost 0.028 \
    --session {opal-chat-session-iri} \
    --escalation deepseek-v4-flash --escalation codestral-latest

• Outcome-level only — task, tier, policy, model, score, latency, cost, escalation events. Prompts, message content, and secrets are NEVER recorded (preferences.ttl Step 36). Traces are private by design: written to references/traces/trace-log.ttl, excluded from the published graph, never part of any public upload.
• Each trace is llmr:RoutingTrace (a subclass of llmr:FeedbackRecord) with llmr:tracedTask, llmr:tracedModel, llmr:costTierUsed, llmr:policyUsed, llmr:qualityScore, llmr:observedLatency, llmr:costIncurred, llmr:escalationEvent, and prov:wasInformedBy the source session.

Then harvest traces into seed-profile refinements and rebuild:

python3 scripts/harvest_traces.py            # report proposed refinements
python3 scripts/harvest_traces.py --apply    # apply + rebuild + GATE

Harvest aggregates observed quality per (model, task), compares against the seed-implied capability score, and — when the deviation exceeds --delta (default 1.0) with at least --min-samples (default 2) — proposes an explicit_overrides adjustment. --apply writes the overrides, rebuilds the graph, and runs the GATE. This is the mechanical half of "feedback prevents stale profiles": traces are the evidence, the seed is the hypothesis, the rebuild produces the corrected frontier.

7b. Manual feedback records (explicit)

1. After each routed execution, append to references/feedback-log.ttl: llmr:FeedbackRecord with model, task, cost tier, quality score, latency, and pass/fail.
2. Review records that contradict the seed profile (e.g., a model scored L4 in practice but the seed says L2). Adjust the matching family_rules entry (or add an explicit_overrides entry) in capability-profiles.json.
3. Rebuild + re-validate (§3).
4. The Pareto frontier, cost tiers, and escalation ladders recompute — routing now reflects reality instead of stale assumptions.

Feedback log template (append a new record per execution):

@prefix llmr: <https://www.openlinksw.com/ontology/llm-routing#> .
@prefix llmrm: <https://www.openlinksw.com/ontology/llm-routing/models/> .
@prefix llmrt: <https://www.openlinksw.com/ontology/llm-routing/tasks/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<https://www.openlinksw.com/ontology/llm-routing/feedback/2026-08-19-0001>
    a llmr:FeedbackRecord ;
    llmr:feedbackModel llmrm:deepseek-v4-flash ;
    llmr:feedbackTask llmrt:code-generation ;
    llmr:qualityScore "4"^^xsd:integer ;
    llmr:costTier "medium"@en ;
    llmr:passedVerification true ;
    llmr:observedLatency "low"@en ;
    llmr:feedbackDate "2026-08-19"^^xsd:date .

8. Anti-Drift Protocol

1. Re-read before build. Before generating any routing output, re-read this SKILL.md's relevant section and the current graph state — never route from memory of a previous session.
2. Gate-first. validate_graph.py is the blocking gate (GATE: 0 failures required). Run it after every rebuild and before any routing decision that uses the graph. A gate run only after delivery is a post-mortem.
3. Section-by-section. Validate the graph before writing recommendations.

9. Post-Generation Checklist

• python3 scripts/validate_graph.py → GATE PASSED: 0 failures
• Graph updated_at within --max-age-days of today
• Every recommended model exists in the graph and meets the task's required_level within the chosen cost tier
• Escalation ladder present and strictly increasing in capability (or cost)
• Governance applied: no blocked model, no residency violation
• Feedback recorded for any executed routing decision

10. Files

llm-routing-skill/
├── SKILL.md                      # this contract
├── README.md                     # quick start
├── CHANGELOG.md
├── references/
│   ├── capability-profiles.json  # curated seed profiles (EDIT ME)
│   ├── task-types.json           # ~30 task taxonomy (EDIT ME)
│   ├── llm-routing-ontology.ttl  # authoritative llmr: TBox (term definitions)
│   ├── routing-graph.ttl         # generated RDF graph (TBox embedded, query target)
│   ├── routing-graph.json        # generated JSON mirror
│   ├── feedback-log.ttl          # manual feedback records (append here)
│   └── traces/trace-log.ttl      # PRIVATE session traces (created by record_trace.py)
├── scripts/
│   ├── fetch_prices.py           # pull live llm-prices.com feeds
│   ├── build_routing_graph.py    # merge prices+profiles+tasks -> graph
│   ├── validate_graph.py         # GATE (blank-node + ontology-term checks)
│   ├── route.py                  # routing decision CLI
│   ├── record_trace.py           # record a private session routing trace
│   └── harvest_traces.py         # traces -> seed refinement -> rebuild
└── examples/
    └── routing-example.md

License

AGPL-3.0