OpenRouter Provider Ranking
Rank the endpoint providers of a single OpenRouter model slug for a
specific workload. Never call the order "globally optimal": it is optimal only
against the stated constraints, weights, token profile and available telemetry.
Mandatory rules
- Apply hard constraints first, score second. Never let high TPS or a low price compensate for an incompatibility.
- For tool calling default to
native-exacto: the :exacto model suffix, no provider.sort, no provider.order. Your own score in that mode is diagnostic, not a replacement for OpenRouter's private telemetry.
- Use
manual only when the user requires a deterministic order, when your own production telemetry outweighs Exacto, or when a failover chain has to be pinned explicitly.
- Never combine
:exacto with provider.sort — an explicit sort wins. Never add provider.order in native-exacto mode.
- Price on the real token profile, not on headline input/output rates. Include cache read/write, per-request fees, conditional pricing overrides and the observed cache hit rate.
- Do not invent missing Exacto, benchmark, cache or tool-success metrics. Apply a conservative prior and an uncertainty penalty, and name the gaps explicitly.
- For multi-turn workloads pass a stable
session_id. Remember that a manual provider.order disables OpenRouter's sticky provider routing, and session_id does not restore it.
- Never store the API key in the skill, a config, a log or the output JSON. Read it only from
OPENROUTER_API_KEY or the environment variable the user names.
- A catalogue field is a hypothesis; a measurement is a fact. Do not hand back a ranking as the answer: always propose the minimal verification of the top candidates against the constraints that were asked for (step 5), and say plainly whether it was run.
- The comparison row is
endpoint × reasoning effort, not the endpoint. Changing effort moves price, latency and quality more than changing provider does — and "none" means "omit the field", not "no reasoning".
- Compute price from components on the token profile (
in, out, cache_read, cache_write, per-request fee). Headline and effective price swap places: the most expensive rate card produced the lowest cost per step at a 99.996 % cache hit.
- Measure cache, latency and reliability on the profile that will ship. Cache hit on a repeated identical prompt (99 %) and in an agent run (68 %) are different numbers; TTFT on a short prompt is 4-6x optimistic; and fitness for a single request does not predict behaviour under sequential load.
Inputs
Collect or estimate:
- the model slug, e.g.
deepseek/deepseek-v4-flash-0731;
uses_tools, streaming, required parameters, context and output limits;
- expected prompt/completion tokens and requests per session;
- cacheable prompt fraction, token-level cache read/write rate, response-cache hit rate;
- hard caps: price, latency, TPS, uptime, quantization, moderation, ZDR/data policy;
- the goal: quality, balanced, interactive latency, cost or batch throughput;
- your own telemetry by provider tag, if any exists.
When the input is incomplete, use the agentic-balanced profile. For tool calling set uses_tools=true. Do not assume a non-zero cache hit rate without observations or a defensible workload model.
Procedure
1. Classify the workload
Pick one profile:
agentic-balanced — the default for agents and B2B SaaS;
agentic-quality — tool correctness and reliability outweigh price;
interactive — minimise TTFT and end-to-end latency;
cost — minimise expected cost under an SLO;
batch — throughput and cost for long completions.
For a goal outside these, override weights; the sum after normalisation must be positive. The formula: references/scoring.md.
2. Fetch fresh endpoint metrics
Preferred path — the OpenRouter Endpoints API through the bundled script:
python3 scripts/rank_providers.py \
--model deepseek/deepseek-v4-flash-0731 \
--config assets/config.example.json \
--format markdown \
--output recommendation.md
For a reproducible or offline analysis:
python3 scripts/rank_providers.py \
--endpoints-file endpoints.json \
--config config.json \
--observations telemetry.jsonl \
--previous-ranking previous-result.json \
--output result.json
The Endpoints API gives the provider tag, pricing, quantization, context and output limits, supported parameters, uptime, and latency and throughput percentiles. When reading a performance page, carry any provider-specific Auto Exacto or benchmark values into observations by exact tag; never guess the match from a display name when a tag exists.
Input formats: references/input-formats.md.
Read the metrics as they are defined, not as they read. latency_* arrives
in milliseconds — 1624.8 is 1.6 s. A p90 throughput is the fast tail and
a p90 latency is the slow one, so rank on p50 and report both. The
storefront's own numbers are collected on short prompts; see step 5 and
references/storefront-traps.md.
3. Add production telemetry
Quality signals, in order of priority:
- your own tool-call/schema success for the same model, prompt class and provider tag;
- provider-specific Exacto or benchmark values from the performance page;
- the endpoint performance/uptime API;
- a conservative prior when nothing is available.
Collect at least: provider_name or provider_tag, success, tool success, prompt/completion/cached/cache-write tokens, TTFT, generation time or TPS, and total cost. For rates from a small sample use a Wilson lower bound, not the raw percentage.
4. Run the ranking and check the result
The script should:
- drop inactive and incompatible endpoints;
- resolve pricing overrides as of the request;
- compute the expected cost for the workload;
- blend OpenRouter percentiles with your own observations by sample confidence;
- normalise cost, TPS, TTFT and E2E against the current eligible pool;
- apply the quality, reliability, cache, fidelity and uncertainty components;
- stabilise the order against a previous result;
- pick a fallback chain with provider-family diversity when the score gap allows.
Check the exit code. On 4 do not relax anything silently: show which constraints eliminated every endpoint and propose the smallest relaxation.
5. Verify the candidates with real requests — mandatory
A ranking is built from catalogue fields, and several of them are wrong often
enough to change the answer: supports_implicit_caching reads false on an
endpoint that caches 99 % of the prefix; storefront TTFT is taken on short
prompts and is 4-6x optimistic; and the cheapest endpoint on a bench may serve
no request at all under sequential load. The full list, each with its
reproduction: references/storefront-traps.md.
So never hand back a ranking as the final answer. Always propose the
minimal verification of the top candidates against the same hard constraints
the score used, and run it where you can:
python3 scripts/probe_endpoints.py \
--model deepseek/deepseek-v4-flash-0731 \
--providers coreweave/fp8,fireworks,siliconflow/fp8 \
--prompt-file real-prompt.txt \
--max-tokens 600 --runs 3 \
--min-tps 50 --max-ttft 6 --require-cache --require-max-tokens
The script sends --runs identical requests per endpoint and reports the
measured TTFT, TPS, cost, cache hit share, completion tokens and error class,
then judges each against the thresholds you pass. Exit code 4 means none
survived — the ranker's own convention; do not relax the thresholds silently.
What must be passed:
--prompt-file with your own prompt once a decision is actually being
made. The --prompt-tokens filler is fine for a rough cut, but TTFT depends
on size and contract compliance depends on content.
--runs >= 2, but do not read a cache miss from filler as a verdict: a fresh
prefix does not become cacheable inside a short probe. The same endpoint gave
99 % over two calls on a prompt that was already in use and 0 % over four on a
new one. A hit proves caching; a miss proves it only with --prompt-file.
- Thresholds taken from the hard constraints, not invented: the check must
answer the user's question, not a generic one.
What the check does not do: it does not measure exact throughput and it does
not certify fitness. A failure is a conclusion; a pass is "on this profile, in
this window, over N calls". For an agentic workload add a run of sequential
calls — fitness for a single request does not predict it.
A separate one-request check of the data policy, whenever the constraints carry
data_collection: deny or zdr: max_tokens: 1, allow_fallbacks: false, one
request per candidate. Distinguish a policy refusal (404 No endpoints found matching your data policy) from a provider 429 — different causes, different
conclusions.
6. Choose the routing mode
| Condition |
Mode |
What to send |
| Tool calls, no fixed order required |
native-exacto |
model: <slug>:exacto, filters/preferences, no sort/order |
| A pinned failover chain, or strong first-party telemetry |
manual |
provider.order from the ranking |
| One simple goal with no custom score |
native OpenRouter |
provider.sort: price/throughput/latency; the bundled ranker is optional |
OpenRouter-specific interactions and limits: references/openrouter-routing.md.
Response format
Always return:
- the chosen mode and a short justification;
- the eligible ranking table: provider tag, score, expected cost per request, TPS percentile, TTFT, E2E, uptime, quality confidence and cache hit rate;
- a ready JSON request fragment;
- the excluded providers, each with its reason;
- coverage and warnings, and the list of missing signals;
- the minimal verification plan — a ready
probe_endpoints.py command with
thresholds taken from the hard constraints, and its result if it was run.
Call the ranking a hypothesis until that check has been made;
- the refresh rule: recompute after noticeable drift, a price or endpoint
change, or enough new telemetry. Never pin an order indefinitely.
In native-exacto, keep the diagnostic ranking and the authoritative runtime ordering by Exacto clearly apart.
Validation checklist
Before returning a result, check that:
- provider tags come from a fresh API call or file, not from memory;
- hard constraints were applied before the score;
- prices were converted from USD/token to readable USD/M and to workload cost without double conversion;
p90 throughput is read as the fast tail — the value about 10 % of requests exceed — and p90 latency as the slow one. Rank on p50; a provider whose p90 is 57 and p50 is 33 is not the fastest, and reading p90 as a floor already put one first;
latency_* is read as milliseconds, while provider.max_price is passed in USD per million tokens and endpoint pricing is stored in USD per token;
provider.order is absent in native-exacto;
- service-tier tags (
/fast, /flex) were not included by accident;
- cache-heavy sessions carry a stable
session_id, and the consequences of a manual order are noted;
- a small sample is not over-read;
- the final request JSON is syntactically valid;
- the minimal verification of the top candidates was proposed — and run where possible — and a ranking without it was called a hypothesis;
supports_implicit_caching, TTFT and throughput were not taken on faith wherever the conclusion depends on them.
Bundled resources
scripts/rank_providers.py — the standalone ranker and routing-fragment generator.
scripts/probe_endpoints.py — the minimal verification of candidates with real
requests: TTFT, TPS, cost, cache hit, max_tokens compliance and error class;
judges each against the thresholds passed, exit code 4 when none survives.
scripts/validate_skill.py — self-check of frontmatter, resources, syntax and unit tests.
assets/config.example.json — a config for a tool-using agent workload.
assets/observations.example.json — aggregated provider observations.
assets/telemetry.example.jsonl — raw request-level observations.
tests/trigger-evals.json — positive and negative activation queries for the description.
references/scoring.md — the formula, normalisation and profiles.
references/input-formats.md — the config and telemetry schema.
references/openrouter-routing.md — semantics OpenRouter routing/caching.
references/storefront-traps.md — reproduced disagreements between catalogue
fields and endpoint behaviour; read it before ranking.
To validate the package, run:
python3 scripts/validate_skill.py
1---2name: openrouter-provider-ranking3description: "Use this skill when a user asks to rank, compare, benchmark, prioritize, or generate routing for OpenRouter provider endpoints (provider.order, provider.only, :exacto) by TPS/throughput, TTFT/latency, effective price, uptime, cache hit rate, tool-call/Exacto quality, quantization, context, privacy, or fallback diversity; also for requests about OpenRouter provider sorting or prioritization. Do not use for broad model-family selection unless endpoint-level provider routing is required."4compatibility: "Requires Python 3.10+; live endpoint discovery requires outbound HTTPS and a management-capable OpenRouter key in OPENROUTER_API_KEY. Offline endpoint JSON is supported. The bundled ranker uses only the Python standard library."5metadata:6 source: https://skillmd.com/skills/codealive-ai-agents-reflection-skills/openrouter-provider-ranking7 version: "1.0.0"8 domain: "openrouter-routing"9---1011# OpenRouter Provider Ranking1213Rank the **endpoint providers of a single OpenRouter model slug** for a14specific workload. Never call the order "globally optimal": it is optimal only15against the stated constraints, weights, token profile and available telemetry.1617## Mandatory rules18191. Apply hard constraints first, score second. Never let high TPS or a low price compensate for an incompatibility.202. For tool calling default to `native-exacto`: the `:exacto` model suffix, no `provider.sort`, no `provider.order`. Your own score in that mode is diagnostic, not a replacement for OpenRouter's private telemetry.213. Use `manual` only when the user requires a deterministic order, when your own production telemetry outweighs Exacto, or when a failover chain has to be pinned explicitly.224. Never combine `:exacto` with `provider.sort` — an explicit sort wins. Never add `provider.order` in `native-exacto` mode.235. Price **on the real token profile**, not on headline input/output rates. Include cache read/write, per-request fees, conditional pricing overrides and the observed cache hit rate.246. Do not invent missing Exacto, benchmark, cache or tool-success metrics. Apply a conservative prior and an uncertainty penalty, and name the gaps explicitly.257. For multi-turn workloads pass a stable `session_id`. Remember that a manual `provider.order` disables OpenRouter's sticky provider routing, and `session_id` does not restore it.268. Never store the API key in the skill, a config, a log or the output JSON. Read it only from `OPENROUTER_API_KEY` or the environment variable the user names.279. **A catalogue field is a hypothesis; a measurement is a fact.** Do not hand back a ranking as the answer: always propose the minimal verification of the top candidates against the constraints that were asked for (step 5), and say plainly whether it was run.2810. The comparison row is `endpoint × reasoning effort`, not the endpoint. Changing effort moves price, latency and quality more than changing provider does — and `"none"` means "omit the field", not "no reasoning".2911. Compute price from components on the token profile (`in`, `out`, `cache_read`, `cache_write`, per-request fee). Headline and effective price swap places: the most expensive rate card produced the lowest cost per step at a 99.996 % cache hit.3012. Measure cache, latency and reliability on the profile that will ship. Cache hit on a repeated identical prompt (99 %) and in an agent run (68 %) are different numbers; TTFT on a short prompt is 4-6x optimistic; and fitness for a single request does not predict behaviour under sequential load.3132## Inputs3334Collect or estimate:3536- the model slug, e.g. `deepseek/deepseek-v4-flash-0731`;37- `uses_tools`, streaming, required parameters, context and output limits;38- expected prompt/completion tokens and requests per session;39- cacheable prompt fraction, token-level cache read/write rate, response-cache hit rate;40- hard caps: price, latency, TPS, uptime, quantization, moderation, ZDR/data policy;41- the goal: quality, balanced, interactive latency, cost or batch throughput;42- your own telemetry by provider tag, if any exists.4344When the input is incomplete, use the `agentic-balanced` profile. For tool calling set `uses_tools=true`. Do not assume a non-zero cache hit rate without observations or a defensible workload model.4546## Procedure4748### 1. Classify the workload4950Pick one profile:5152- `agentic-balanced` — the default for agents and B2B SaaS;53- `agentic-quality` — tool correctness and reliability outweigh price;54- `interactive` — minimise TTFT and end-to-end latency;55- `cost` — minimise expected cost under an SLO;56- `batch` — throughput and cost for long completions.5758For a goal outside these, override `weights`; the sum after normalisation must be positive. The formula: [references/scoring.md](references/scoring.md).5960### 2. Fetch fresh endpoint metrics6162Preferred path — the OpenRouter Endpoints API through the bundled script:6364```bash65python3 scripts/rank_providers.py \66 --model deepseek/deepseek-v4-flash-0731 \67 --config assets/config.example.json \68 --format markdown \69 --output recommendation.md70```7172For a reproducible or offline analysis:7374```bash75python3 scripts/rank_providers.py \76 --endpoints-file endpoints.json \77 --config config.json \78 --observations telemetry.jsonl \79 --previous-ranking previous-result.json \80 --output result.json81```8283The Endpoints API gives the provider tag, pricing, quantization, context and output limits, supported parameters, uptime, and latency and throughput percentiles. When reading a performance page, carry any provider-specific Auto Exacto or benchmark values into observations by exact `tag`; never guess the match from a display name when a tag exists.8485Input formats: [references/input-formats.md](references/input-formats.md).8687**Read the metrics as they are defined, not as they read.** `latency_*` arrives88in **milliseconds** — `1624.8` is 1.6 s. A p90 *throughput* is the fast tail and89a p90 *latency* is the slow one, so rank on `p50` and report both. The90storefront's own numbers are collected on short prompts; see step 5 and91[references/storefront-traps.md](references/storefront-traps.md).9293### 3. Add production telemetry9495Quality signals, in order of priority:96971. your own tool-call/schema success for the same model, prompt class and provider tag;982. provider-specific Exacto or benchmark values from the performance page;993. the endpoint performance/uptime API;1004. a conservative prior when nothing is available.101102Collect at least: `provider_name` or `provider_tag`, success, tool success, prompt/completion/cached/cache-write tokens, TTFT, generation time or TPS, and total cost. For rates from a small sample use a Wilson lower bound, not the raw percentage.103104### 4. Run the ranking and check the result105106The script should:107108- drop inactive and incompatible endpoints;109- resolve pricing overrides as of the request;110- compute the expected cost for the workload;111- blend OpenRouter percentiles with your own observations by sample confidence;112- normalise cost, TPS, TTFT and E2E against the current eligible pool;113- apply the quality, reliability, cache, fidelity and uncertainty components;114- stabilise the order against a previous result;115- pick a fallback chain with provider-family diversity when the score gap allows.116117Check the exit code. On `4` do not relax anything silently: show which constraints eliminated every endpoint and propose the smallest relaxation.118119### 5. Verify the candidates with real requests — mandatory120121A ranking is built from catalogue fields, and several of them are wrong often122enough to change the answer: `supports_implicit_caching` reads `false` on an123endpoint that caches 99 % of the prefix; storefront TTFT is taken on short124prompts and is 4-6x optimistic; and the cheapest endpoint on a bench may serve125no request at all under sequential load. The full list, each with its126reproduction: [references/storefront-traps.md](references/storefront-traps.md).127128So **never hand back a ranking as the final answer**. Always propose the129minimal verification of the top candidates against the same hard constraints130the score used, and run it where you can:131132```bash133python3 scripts/probe_endpoints.py \134 --model deepseek/deepseek-v4-flash-0731 \135 --providers coreweave/fp8,fireworks,siliconflow/fp8 \136 --prompt-file real-prompt.txt \137 --max-tokens 600 --runs 3 \138 --min-tps 50 --max-ttft 6 --require-cache --require-max-tokens139```140141The script sends `--runs` identical requests per endpoint and reports the142measured TTFT, TPS, cost, cache hit share, completion tokens and error class,143then judges each against the thresholds you pass. Exit code `4` means none144survived — the ranker's own convention; do not relax the thresholds silently.145146What must be passed:147148- `--prompt-file` with **your own** prompt once a decision is actually being149 made. The `--prompt-tokens` filler is fine for a rough cut, but TTFT depends150 on size and contract compliance depends on content.151- `--runs >= 2`, but do not read a cache miss from filler as a verdict: a fresh152 prefix does not become cacheable inside a short probe. The same endpoint gave153 99 % over two calls on a prompt that was already in use and 0 % over four on a154 new one. A hit proves caching; a miss proves it only with `--prompt-file`.155- Thresholds taken from the hard constraints, not invented: the check must156 answer the user's question, not a generic one.157158What the check does **not** do: it does not measure exact throughput and it does159not certify fitness. A failure is a conclusion; a pass is "on this profile, in160this window, over N calls". For an agentic workload add a run of sequential161calls — fitness for a single request does not predict it.162163A separate one-request check of the data policy, whenever the constraints carry164`data_collection: deny` or `zdr`: `max_tokens: 1`, `allow_fallbacks: false`, one165request per candidate. Distinguish a policy refusal (`404 No endpoints found166matching your data policy`) from a provider 429 — different causes, different167conclusions.168169### 6. Choose the routing mode170171| Condition | Mode | What to send |172|---|---|---|173| Tool calls, no fixed order required | `native-exacto` | `model: <slug>:exacto`, filters/preferences, no `sort`/`order` |174| A pinned failover chain, or strong first-party telemetry | `manual` | `provider.order` from the ranking |175| One simple goal with no custom score | native OpenRouter | `provider.sort: price/throughput/latency`; the bundled ranker is optional |176177OpenRouter-specific interactions and limits: [references/openrouter-routing.md](references/openrouter-routing.md).178179## Response format180181Always return:1821831. the chosen mode and a short justification;1842. the eligible ranking table: provider tag, score, expected cost per request, TPS percentile, TTFT, E2E, uptime, quality confidence and cache hit rate;1853. a ready JSON request fragment;1864. the excluded providers, each with its reason;1875. coverage and warnings, and the list of missing signals;1886. **the minimal verification plan** — a ready `probe_endpoints.py` command with189 thresholds taken from the hard constraints, and its result if it was run.190 Call the ranking a hypothesis until that check has been made;1917. the refresh rule: recompute after noticeable drift, a price or endpoint192 change, or enough new telemetry. Never pin an order indefinitely.193194In `native-exacto`, keep **the diagnostic ranking** and **the authoritative runtime ordering by Exacto** clearly apart.195196## Validation checklist197198Before returning a result, check that:199200- provider tags come from a fresh API call or file, not from memory;201- hard constraints were applied before the score;202- prices were converted from USD/token to readable USD/M and to workload cost without double conversion;203- **`p90` throughput is read as the fast tail — the value about 10 % of requests exceed — and `p90` latency as the slow one. Rank on `p50`;** a provider whose p90 is 57 and p50 is 33 is not the fastest, and reading p90 as a floor already put one first;204- `latency_*` is read as milliseconds, while `provider.max_price` is passed in USD per million tokens and endpoint pricing is stored in USD per token;205- `provider.order` is absent in `native-exacto`;206- service-tier tags (`/fast`, `/flex`) were not included by accident;207- cache-heavy sessions carry a stable `session_id`, and the consequences of a manual order are noted;208- a small sample is not over-read;209- the final request JSON is syntactically valid;210- the minimal verification of the top candidates was proposed — and run where possible — and a ranking without it was called a hypothesis;211- `supports_implicit_caching`, TTFT and throughput were not taken on faith wherever the conclusion depends on them.212213## Bundled resources214215- `scripts/rank_providers.py` — the standalone ranker and routing-fragment generator.216- `scripts/probe_endpoints.py` — the minimal verification of candidates with real217 requests: TTFT, TPS, cost, cache hit, `max_tokens` compliance and error class;218 judges each against the thresholds passed, exit code `4` when none survives.219- `scripts/validate_skill.py` — self-check of frontmatter, resources, syntax and unit tests.220- `assets/config.example.json` — a config for a tool-using agent workload.221- `assets/observations.example.json` — aggregated provider observations.222- `assets/telemetry.example.jsonl` — raw request-level observations.223- `tests/trigger-evals.json` — positive and negative activation queries for the description.224- `references/scoring.md` — the formula, normalisation and profiles.225- `references/input-formats.md` — the config and telemetry schema.226- `references/openrouter-routing.md` — semantics OpenRouter routing/caching.227- `references/storefront-traps.md` — reproduced disagreements between catalogue228 fields and endpoint behaviour; read it before ranking.229230To validate the package, run:231232```bash233python3 scripts/validate_skill.py234```