skill-triage
Skill routing fails quietly. With dozens of skill descriptions loaded at once, a
request lands on a plausible-but-wrong skill, or on none, and nobody notices
until the output is off. This skill makes routing measurable: it asks a
small decision model which skill fits, gets back a probability for every skill,
and turns that into accuracy numbers and a list of who is stealing from whom.
What Jev is (and isn't)
Jev is TypeSafe AI's "System One" model. You send it a state (here, the user
request) plus typed questions. It returns typed answers: a choice over up to
255 options with a full probability distribution and a confidence, or a noul
(the probability a statement is true). It does not write text or code, and it
cannot replace the model running your session. It is a fast, cheap classifier
that you call from code.
That shape fits skill routing well. Each skill's name and description become
one option of a Choice question, which is roughly the same text a host sees
when it decides whether to load a skill.
TypeSafe ships its own agent skill (typesafe-ai/skills) that teaches agents
the API in general. This skill doesn't duplicate it. It applies Jev to one job:
routing this catalog.
The script
Everything runs through scripts/triage.py (python3 stdlib, no install).
It discovers skills the same way scripts/catalog.sh does
(skills/<category>/<skill>/SKILL.md plus the top-level orchestrator, skipping
archive/, in-progress/, node_modules/). Pass --root for any other
skills directory, and --only to narrow the catalog to the skills actually
available on this host.
| Mode |
What it answers |
Calls |
route "<request>" |
Which one skill should handle this, or none |
2 |
check --cases F / --from-routing-table F |
How often the catalog routes labeled requests correctly |
1 per 32 cases |
collisions [--skill X] |
Which skills lose their own trigger phrases, and to whom |
1 per 32 trigger phrases |
check and collisions batch: the catalog goes in the state once per call and
each probe is one Choice question over the skill names, up to 32 per call
(Venice rejects 40). --batch-size 1 restores one call per probe, which costs
about 6× more and scored no better (62/65 vs 64/65 on the routing table).
Every run has a spend cap. The script estimates the cost before calling and
refuses anything over --max-cost (default $0.02, or JEV_MAX_COST) with exit
2. If the real spend still reaches the cap mid-run, it stops, prints the
partial results, and exits 5. Use --dry-run to see the estimate, and scope
with --skill rather than raising the cap: a full-catalog collision scan
(about $0.036) is refused by default on purpose.
How route decides
It follows TypeSafe's published skill-suggestion pattern (which cut wrong skill
loads from 16.8% to 7.3% over 488 requests in their test):
- Wide rank. One Choice over every skill, plus three gate nouls: does the
request act on the user's system, would an expert follow a documented
procedure, could a short answer suffice (inverted). If the gate mean is below
GATE_THRESHOLD, the answer is no skill. That is a legitimate result,
the same as madness's "none is a legal outcome."
- Rerank. The top
SHORTLIST skills get their description plus a body
excerpt, a second Choice, and a per-skill "fits" noul. If no candidate fits
above FITS_THRESHOLD, there's no winner.
The thresholds are constants at the top of the script, kept in one place so a
human can review and tune them. Change them there, not with ad-hoc flags.
Why check and collisions rank on descriptions only
The host decides whether to load a skill from its name and description, not
its body. Scoring that same text means a miss points at the description,
which is the thing you can fix. route is different: it's making a real
decision, so it uses every signal it has.
Credentials, cost, privacy
Key: Venice is the default provider. The script uses the first of
VENICE_API_KEY (api.venice.ai/api), TYPESAFE_API_KEY (direct,
api.typesafe.ai) or OPENROUTER_API_KEY that is set, and JEV_PROVIDER
forces one. All three serve the same /v1/systemone contract. Venice omits usage.cost, so the script estimates it
from Venice's published input price, and Venice only offers jev-latest, so
pinned before/after comparisons need TypeSafe or OpenRouter. Look for
the key name in the environment and the project's .env files before
asking the user for anything, and pass --env-file rather than exporting it.
The script never prints the value.
Model: JEV_MODEL/--model, default jev-latest. Every run reports the
model that actually served it. Pin a version (e.g. jev-1.13) when comparing
two check runs, or a model change will look like a description change.
Cost: measured 2026-10-25 via Venice on the 78-skill catalog:
- a
route is about $0.0006 and takes under a second
- the 65-case routing-table
check is $0.0054 (3 calls)
collisions --skill X is about $0.001
- Jev bills about 1,300 input tokens per batched question, whatever the
option labels, so roughly $0.00005 per probe is the floor.
A 402
"insufficient credits" means the account needs credits; topping up is the
user's call, never yours.
Fallback (exit 4): no key, a rejected key, an empty account (401/402/403),
an unreachable endpoint, or rate limits that outlast the retries all exit
4 with a one-line jev unavailable: ... reason. Exit 4 means "carry on
without Jev": route by judgment exactly as you would have, and say so in one
line. Never retry in a loop and never fabricate probabilities. Exit 3 is
different: the API rejected the request itself, which is a bug to report.
Privacy: every call sends the request text and the catalog's
descriptions to the provider in use: Venice by default, or TypeSafe or
OpenRouter. The catalog here
is public OSS, but a user's request may not be. Don't route private requests
through it unless the user has opted in.
Reading the results
Treat Jev as a fast second opinion, not the judge. A disagreement is a reason
to look, and either side can be wrong:
- Stealers (
winner <- owner: count). The winner's description claims
territory the owner meant to own. Usually you'd sharpen the owner's
phrasing or add a scoped "not for X" to the stealer. Hand the edit to
skill-update, then re-run the same check to confirm it moved.
- The label may be wrong. Example from the first run: "run until green" went
to
loop-controller over fix-until-green, but madness deliberately routes
loops through loop-controller. That's a routing-table question, not a
description bug. Say which one you think it is, and why.
- Probe noise in
collisions. Probes are the quoted strings after the
description's first trigger marker ("Trigger on", "Use when", ...), or every
quote if there's no marker. A quote that's prose, not a trigger (e.g. "don't
skip it because it 'looks fine'"), shows up as a miss. Read the quote in
context before calling it a collision.
- Thin margins (correct, but by less than
LOW_MARGIN). These are the next
misses waiting to happen, so they're worth fixing before they flip.
- Skipped phrases in
--from-routing-table point at skills outside this
catalog (plugin skills like superpowers:*). That's expected; to include
them, add their roots with --root.
For a report, give the accuracy, the stealer list, each miss with a one-line
verdict (description fix / label fix / genuine ambiguity), and the served model
and cost.
Composing with the rest of the toolkit
- madness: the pre-check is off by default, because every call costs money
and sends the request off the machine. It runs only when
SKILL_TRIAGE_ROUTE=1
is set or the user asks for Jev in the session. madness then runs route and
uses the result as a hint; its own judgment and its confirm gate still decide.
A Jev pick never skips the "...go?" on an expensive route. Any non-zero exit
means madness routes as if the pre-check never ran.
- skill-review / skill-update:
collisions --skill <name> gives trigger
evidence for one skill under review. Re-run it after the update.
- skill-writer: after adding a skill, run
collisions --skill <new> and a
check over the routing table. A new skill's commonest failure is stealing
an existing skill's triggers.
When not to use it
- The user already named the skill. Run that skill.
- The script exits 4 (no key, empty account, no network). Say so in one line
and route by judgment. Never fabricate probabilities.
- Grading skill output quality. This measures which skill is picked, not how
well it works once loaded. That's the skill-creator eval loop.
Anti-patterns
| Anti-pattern |
Why it fails |
| Rewriting a description until Jev agrees |
Overfits one classifier; the real router is the host model. Fix only what you can explain in a sentence |
Comparing runs on jev-latest |
The alias moves (1.13 → 1.14 in October 2026); pin the model for before/after |
Raising --max-cost to push a full scan through |
The cap is the user's budget; scope with --skill to the skills you changed instead |
| Letting a route hint override madness's confirm gate |
Cost and blast radius are madness's call, not the classifier's |
| Sending private requests by default |
Every call leaves the machine; opt-in only |
1---2name: skill-triage3version: 0.2.24description: >-5 Measure skill routing with TypeSafe's Jev decision model: which skill a6 request should load, accuracy against labeled routing cases, and description7 collisions where one skill steals another's triggers. Cheap and spend-capped.8 Use when skills misfire or under-trigger, or before and after editing9 descriptions. Trigger on "jev", "check routing accuracy", "description10 collisions", "trigger check".11requires_agent_teams: false12requires_claude_code: false13min_plan: starter14owns:15 directories: []16 patterns: []17 shared_read: ["skills/"]18allowed-tools: ["Read", "Bash", "Grep", "Glob"]19composes_with: ["madness", "skill-review", "skill-update", "skill-explorer", "skill-writer"]20spawned_by: []21metadata:22 source: https://skillmd.com/skills/ivy00johns/skill-triage23---2425# skill-triage2627Skill routing fails quietly. With dozens of skill descriptions loaded at once, a28request lands on a plausible-but-wrong skill, or on none, and nobody notices29until the output is off. This skill makes routing **measurable**: it asks a30small decision model which skill fits, gets back a probability for every skill,31and turns that into accuracy numbers and a list of who is stealing from whom.3233## What Jev is (and isn't)3435Jev is TypeSafe AI's "System One" model. You send it a _state_ (here, the user36request) plus typed questions. It returns typed answers: a `choice` over up to37255 options with a full probability distribution and a confidence, or a `noul`38(the probability a statement is true). It does not write text or code, and it39cannot replace the model running your session. It is a fast, cheap classifier40that you call from code.4142That shape fits skill routing well. Each skill's name and description become43one option of a Choice question, which is roughly the same text a host sees44when it decides whether to load a skill.4546TypeSafe ships its own agent skill (`typesafe-ai/skills`) that teaches agents47the API in general. This skill doesn't duplicate it. It applies Jev to one job:48routing this catalog.4950## The script5152Everything runs through `scripts/triage.py` (python3 stdlib, no install).53It discovers skills the same way `scripts/catalog.sh` does54(`skills/<category>/<skill>/SKILL.md` plus the top-level orchestrator, skipping55`archive/`, `in-progress/`, `node_modules/`). Pass `--root` for any other56skills directory, and `--only` to narrow the catalog to the skills actually57available on this host.5859| Mode | What it answers | Calls |60| -------------------------------------------- | ---------------------------------------------------------- | --------------------------- |61| `route "<request>"` | Which one skill should handle this, or none | 2 |62| `check --cases F` / `--from-routing-table F` | How often the catalog routes labeled requests correctly | 1 per 32 cases |63| `collisions [--skill X]` | Which skills lose their _own_ trigger phrases, and to whom | 1 per 32 trigger phrases |6465`check` and `collisions` batch: the catalog goes in the state once per call and66each probe is one Choice question over the skill names, up to 32 per call67(Venice rejects 40). `--batch-size 1` restores one call per probe, which costs68about 6× more and scored no better (62/65 vs 64/65 on the routing table).6970**Every run has a spend cap.** The script estimates the cost before calling and71refuses anything over `--max-cost` (default $0.02, or `JEV_MAX_COST`) with exit722. If the real spend still reaches the cap mid-run, it stops, prints the73partial results, and exits 5. Use `--dry-run` to see the estimate, and scope74with `--skill` rather than raising the cap: a full-catalog collision scan75(about $0.036) is refused by default on purpose.7677### How `route` decides7879It follows TypeSafe's published skill-suggestion pattern (which cut wrong skill80loads from 16.8% to 7.3% over 488 requests in their test):81821. **Wide rank.** One Choice over every skill, plus three gate nouls: does the83 request act on the user's system, would an expert follow a documented84 procedure, could a short answer suffice (inverted). If the gate mean is below85 `GATE_THRESHOLD`, the answer is **no skill**. That is a legitimate result,86 the same as madness's "none is a legal outcome."872. **Rerank.** The top `SHORTLIST` skills get their description plus a body88 excerpt, a second Choice, and a per-skill "fits" noul. If no candidate fits89 above `FITS_THRESHOLD`, there's no winner.9091The thresholds are constants at the top of the script, kept in one place so a92human can review and tune them. Change them there, not with ad-hoc flags.9394### Why `check` and `collisions` rank on descriptions only9596The host decides whether to load a skill from its name and description, not97its body. Scoring that same text means a miss points at the _description_,98which is the thing you can fix. `route` is different: it's making a real99decision, so it uses every signal it has.100101## Credentials, cost, privacy102103- **Key:** Venice is the default provider. The script uses the first of104 `VENICE_API_KEY` (`api.venice.ai/api`), `TYPESAFE_API_KEY` (direct,105 `api.typesafe.ai`) or `OPENROUTER_API_KEY` that is set, and `JEV_PROVIDER`106 forces one. All three serve the same `/v1/systemone` contract. Venice omits `usage.cost`, so the script estimates it107 from Venice's published input price, and Venice only offers `jev-latest`, so108 pinned before/after comparisons need TypeSafe or OpenRouter. Look for109 the key _name_ in the environment and the project's `.env` files before110 asking the user for anything, and pass `--env-file` rather than exporting it.111 The script never prints the value.112- **Model:** `JEV_MODEL`/`--model`, default `jev-latest`. Every run reports the113 model that actually served it. Pin a version (e.g. `jev-1.13`) when comparing114 two `check` runs, or a model change will look like a description change.115- **Cost:** measured 2026-10-25 via Venice on the 78-skill catalog:116 - a `route` is about $0.0006 and takes under a second117 - the 65-case routing-table `check` is $0.0054 (3 calls)118 - `collisions --skill X` is about $0.001119 - Jev bills about 1,300 input tokens per batched question, whatever the120 option labels, so roughly $0.00005 per probe is the floor.121122 A 402123 "insufficient credits" means the account needs credits; topping up is the124 user's call, never yours.125- **Fallback (exit 4):** no key, a rejected key, an empty account (401/402/403),126 an unreachable endpoint, or rate limits that outlast the retries all exit127 **4** with a one-line `jev unavailable: ...` reason. Exit 4 means "carry on128 without Jev": route by judgment exactly as you would have, and say so in one129 line. Never retry in a loop and never fabricate probabilities. Exit 3 is130 different: the API rejected the request itself, which is a bug to report.131- **Privacy:** every call sends the request text and the catalog's132 descriptions to the provider in use: Venice by default, or TypeSafe or133 OpenRouter. The catalog here134 is public OSS, but a user's request may not be. Don't route private requests135 through it unless the user has opted in.136137## Reading the results138139Treat Jev as a fast second opinion, not the judge. A disagreement is a reason140to look, and either side can be wrong:141142- **Stealers** (`winner <- owner: count`). The winner's description claims143 territory the owner meant to own. Usually you'd sharpen the _owner's_144 phrasing or add a scoped "not for X" to the stealer. Hand the edit to145 `skill-update`, then re-run the same `check` to confirm it moved.146- **The label may be wrong.** Example from the first run: "run until green" went147 to `loop-controller` over `fix-until-green`, but madness deliberately routes148 loops through `loop-controller`. That's a routing-table question, not a149 description bug. Say which one you think it is, and why.150- **Probe noise in `collisions`.** Probes are the quoted strings after the151 description's first trigger marker ("Trigger on", "Use when", ...), or every152 quote if there's no marker. A quote that's prose, not a trigger (e.g. "don't153 skip it because it 'looks fine'"), shows up as a miss. Read the quote in154 context before calling it a collision.155- **Thin margins** (correct, but by less than `LOW_MARGIN`). These are the next156 misses waiting to happen, so they're worth fixing before they flip.157- **Skipped phrases** in `--from-routing-table` point at skills outside this158 catalog (plugin skills like `superpowers:*`). That's expected; to include159 them, add their roots with `--root`.160161For a report, give the accuracy, the stealer list, each miss with a one-line162verdict (description fix / label fix / genuine ambiguity), and the served model163and cost.164165## Composing with the rest of the toolkit166167- **madness:** the pre-check is off by default, because every call costs money168 and sends the request off the machine. It runs only when `SKILL_TRIAGE_ROUTE=1`169 is set or the user asks for Jev in the session. madness then runs `route` and170 uses the result as a hint; its own judgment and its confirm gate still decide.171 A Jev pick never skips the "...go?" on an expensive route. Any non-zero exit172 means madness routes as if the pre-check never ran.173- **skill-review / skill-update:** `collisions --skill <name>` gives trigger174 evidence for one skill under review. Re-run it after the update.175- **skill-writer:** after adding a skill, run `collisions --skill <new>` and a176 `check` over the routing table. A new skill's commonest failure is stealing177 an existing skill's triggers.178179## When not to use it180181- The user already named the skill. Run that skill.182- The script exits 4 (no key, empty account, no network). Say so in one line183 and route by judgment. Never fabricate probabilities.184- Grading skill _output quality_. This measures which skill is picked, not how185 well it works once loaded. That's the skill-creator eval loop.186187## Anti-patterns188189| Anti-pattern | Why it fails |190| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |191| Rewriting a description until Jev agrees | Overfits one classifier; the real router is the host model. Fix only what you can explain in a sentence |192| Comparing runs on `jev-latest` | The alias moves (1.13 → 1.14 in October 2026); pin the model for before/after |193| Raising `--max-cost` to push a full scan through | The cap is the user's budget; scope with `--skill` to the skills you changed instead |194| Letting a route hint override madness's confirm gate | Cost and blast radius are madness's call, not the classifier's |195| Sending private requests by default | Every call leaves the machine; opt-in only |