Skill Quality Evaluator
Overview
RAG and gateways (the two sibling Ch6 skills) solve the routing problem: which
skill matches this task? They do not solve the quality problem: is this skill
worth trusting once matched? Raw skill libraries accumulate junk. As a
repository scales from dozens to hundreds to thousands of skills, retrieval
quality degrades because the agent pulls low-quality or unsafe skills alongside
the useful ones.
The chapter grounds this in two research anchors:
- SkillsBench (DAIR.AI). Curated skills improve task performance by 16.2%.
Self-generated skills — produced by agents without human review — show zero
improvement over baseline. Quality matters more than quantity, and 2-3
focused skills per task is the optimal number.
- SkillNet (2026). A repository of 200,000+ reusable skills, each rated
across five quality dimensions. Agents using SkillNet-rated skills achieved
40% higher average rewards and 30% fewer execution steps across ALFWorld,
WebShop, and ScienceWorld — held across multiple backbone models, confirming
quality-rated retrieval is a model-agnostic infrastructure layer.
The five dimensions (Table 6-1) and the composite weighting:
| Dimension |
Weight |
Maps to |
| safety |
2.0 |
security scanning |
| completeness |
1.0 |
dependency checking |
| executability |
2.0 |
hallucinated-tool-call risk |
| maintainability |
1.0 |
composability |
| cost_awareness |
1.0 |
token economics (Ch8) |
Composite = weighted mean with safety and executability at double weight, so a
skill that hallucinates tool calls (low executability) or runs unconstrained
shell commands (low safety) ranks low regardless of how well it documents.
When to Use
- A skill library has grown past ~20 entries and mixed provenance (vendor,
team-authored, agent-generated)
- Retrieval is surfacing plausible-but-junk skills alongside good ones
- You need a deployment knob (
min_quality) that encodes organizational risk
appetite (research 0.4 → production healthcare 0.8+)
- You are integrating quality as a second ranking signal on top of relevance
Phrases that invoke this skill: "rate this skill", "quality gate", "SkillNet",
"is this skill safe to use", "why did the agent pick a bad skill".
When NOT to Use
- Routing / matching. Which skill fits the task is
rag-mcp-tool-selection
or the gateway. This skill assumes a matched candidate set and filters it.
- Libraries under ~20 curated skills. The chapter's tip: start with a
curated, quality-gated set (hundreds, not thousands). Below that, hand audit.
- As a security scanner.
safety and executability are heuristic scores
that flag a skill for review; they are not a substitute for agent-shield /
SkillSpector static analysis before install.
- As a self-generation approver. SkillsBench shows self-generated skills
score zero improvement; a passing composite does not license auto-install of
agent-authored skills without human curation.
Process
| Step |
Input |
Action |
Output |
Verification |
| 1 |
Skills catalog JSON (each skill carries eval_safety/completeness/executability/maintainability/cost_awareness) |
lib.load_catalog(path) |
List of skill dicts |
len(catalog) > 0; each skill has all five eval_* keys in [0,1] |
| 2 |
One skill dict |
lib.SkillQuality.from_skill(skill).composite |
Composite float in [0,1] |
Safety+executability weighted 2x; recompute by hand for one skill to confirm /7 mean |
| 3 |
Task description + catalog |
lib.retrieve_quality_gated(task, catalog, min_quality=0.6, top_k=3) |
Top-K skills passing the gate, ranked by relevance*quality |
Hard gate: eval_safety>0 AND eval_executability>0; soft gate: composite >= min_quality |
| 4 |
Same, varying min_quality |
lib.retrieve_quality_gated(..., min_quality=0.8) |
Smaller/empty result |
Raising the threshold monotonically shrinks the result set |
| 5 |
Catalog + a battery of task queries |
lib.monitor_gaps(catalog, queries, min_quality) |
Counts of no_relevant_skill vs low_quality_filtered events |
Interpret: many no-relevant = repository too small; many low-quality-filtered = gate too permissive OR catalog quality poor |
| 6 |
Selected skill |
(your agent runtime) loads the SKILL.md and executes |
Task result |
If the top skill still misbehaves, its eval_* ratings are stale — re-rate |
Rationalizations
| Agent rationalization |
Documented rebuttal |
| "More skills always help — just retrieve top-K by relevance." |
SkillsBench: self-generated (unrated) skills give zero improvement, and 2-3 focused skills per task is optimal. Relevance without a quality gate re-admits the junk that degrades retrieval at scale. |
| "Weight all five dimensions equally — it's simpler." |
The chapter weights safety and executability 2x deliberately: a skill that runs unconstrained shell commands or hallucinates tool calls is dangerous regardless of maintainability. Equal weighting lets a well-documented unsafe skill outrank a plain safe one. |
| "Rank by quality alone — always pick the highest-rated skill." |
The chapter ranks by relevance * quality multiplicatively. Pure quality ranking returns the best skill for the wrong task. Pure relevance returns the junk. The product is the point. |
| "Set min_quality once globally." |
The threshold is a per-deployment risk decision: 0.4 for research to cast a wide net, 0.8+ for production healthcare. One global value ignores that risk appetite differs by system. |
| "A high composite means the skill is safe to auto-install." |
Composite is a retrieval-ranking signal, not an install gate. Self-generated skills need human curation (SkillsBench) and static scanning (agent-shield/SkillSpector) before install regardless of score. |
Red Flags
- Every skill scores above min_quality. The ratings are inflated or the gate
is too low; the monitor will show near-zero
low_quality_filtered events even
on a junk-laden catalog.
- Safety or executability is 0 but the skill still ranks. The hard gate
(
eval_safety>0 AND eval_executability>0) is being bypassed — a zero on either
dimension must exclude the skill entirely, not merely lower it.
- Raising min_quality does not shrink results. The soft gate is not wired;
the threshold knob is inert.
- High
no_relevant_skill AND high low_quality_filtered together. The
repository is both too small and too junky — grow the curated set, do not just
lower the gate.
- CLI
--help exits non-zero. SKILL.md / CLI mismatch; the multi-harness
invariant is broken.
Non-Negotiable Verification
Run the benchmark battery.
python cli.py benchmark
Confirm the gate admits high-quality skills and rejects the seeded
low-quality ones across the task battery, and that raising --min-quality
monotonically shrinks the admitted set.
Hand-check one composite.
python cli.py rate safe-readonly-sql-query
Recompute (safety*2 + completeness + executability*2 + maintainability + cost_awareness) / 7 by hand and confirm it matches.
Prove the hard gate.
python cli.py retrieve "run a shell command to clean up disk" --json
Confirm the seeded unsafe-shell-runner skill (eval_safety = 0) never
appears, even when it is the most textually relevant.
JSON round-trips.
python cli.py retrieve "..." --json | python -c "import json,sys; json.load(sys.stdin)"
Security Posture
- Prompt injection. Skill descriptions and
synthetic_queries are used for
relevance matching; treat them as untrusted if the catalog ingests
community-contributed skills. A malicious skill could inflate its own eval_*
ratings — ratings must come from an independent evaluator, not self-reported by
the skill author. The # TODO(production): seam in lib.score_relevance and
the rating-provenance note in load_catalog mark where to enforce this.
- Data exfiltration. No network calls in
lib.py; catalog is read from an
explicit path.
- Privilege escalation. The
safety dimension is the flag for skills that
execute shell / write files / hold broad credentials. A zero on safety hard-
excludes; do not soften that gate to admit a convenient-but-unsafe skill.
Composition
- Composes after
rag-mcp-tool-selection / mcp-gateway-two-meta-tools:
they return the relevant candidate set; this skill filters that set by quality.
- Feeds graph-based retrieval — the composite maps to the Cypher
relevance * quality DESC ordering in the chapter's Example 6-3.
- Pairs with
agent-shield / SkillSpector at the install boundary (this
skill gates retrieval, those gate installation).
- cost_awareness links directly to the Ch8 token-economics material.
Source Attribution
Distilled from Agentic GraphRAG (O'Reilly), Chapter 6 — Tool Orchestration,
sections "Skills: The Judgment Layer", "Skill quality evaluation", "Integrating
quality ratings into graph-based retrieval", and "Scale and retrieval". Named
references:
- SkillsBench (DAIR.AI) — 16.2% curated-skill improvement; zero for
self-generated; 2-3 focused skills per task optimal
- SkillNet (2026) — 200,000-skill repository; five-dimensional rating; 40%
higher rewards / 30% fewer steps across ALFWorld, WebShop, ScienceWorld
- SkillRL — 10-20x token compression per skill; 7B model at 89.9% on ALFWorld
- Anthropic Agent Skills specification — SKILL.md progressive disclosure
1---2name: skill-quality-evaluator3description: Score a skill against SkillNet's five quality dimensions (safety, completeness, executability, maintainability, cost_awareness), compute a safety/executability-weighted composite, and gate skill retrieval so an agent pulls the most-relevant skill that ALSO clears a quality threshold. Use when a skill library has grown past a few dozen entries and retrieval is surfacing low-quality or unsafe skills alongside useful ones. NOT for routing (which skill matches this task? — that is rag-mcp-tool-selection), NOT for a library under ~20 curated skills (quality is still trivially auditable by hand), NOT a substitute for a security scanner (executability and safety scores are heuristics that flag review, not proofs).4---56# Skill Quality Evaluator78## Overview910RAG and gateways (the two sibling Ch6 skills) solve the routing problem: which11skill matches this task? They do not solve the quality problem: is this skill12worth trusting once matched? Raw skill libraries accumulate junk. As a13repository scales from dozens to hundreds to thousands of skills, retrieval14quality degrades because the agent pulls low-quality or unsafe skills alongside15the useful ones.1617The chapter grounds this in two research anchors:1819- **SkillsBench (DAIR.AI).** Curated skills improve task performance by 16.2%.20 Self-generated skills — produced by agents without human review — show zero21 improvement over baseline. Quality matters more than quantity, and 2-322 focused skills per task is the optimal number.23- **SkillNet (2026).** A repository of 200,000+ reusable skills, each rated24 across five quality dimensions. Agents using SkillNet-rated skills achieved25 40% higher average rewards and 30% fewer execution steps across ALFWorld,26 WebShop, and ScienceWorld — held across multiple backbone models, confirming27 quality-rated retrieval is a model-agnostic infrastructure layer.2829The five dimensions (Table 6-1) and the composite weighting:3031| Dimension | Weight | Maps to |32|-----------|--------|---------|33| safety | 2.0 | security scanning |34| completeness | 1.0 | dependency checking |35| executability | 2.0 | hallucinated-tool-call risk |36| maintainability | 1.0 | composability |37| cost_awareness | 1.0 | token economics (Ch8) |3839Composite = weighted mean with safety and executability at double weight, so a40skill that hallucinates tool calls (low executability) or runs unconstrained41shell commands (low safety) ranks low regardless of how well it documents.4243## When to Use4445- A skill library has grown past ~20 entries and mixed provenance (vendor,46 team-authored, agent-generated)47- Retrieval is surfacing plausible-but-junk skills alongside good ones48- You need a deployment knob (`min_quality`) that encodes organizational risk49 appetite (research 0.4 → production healthcare 0.8+)50- You are integrating quality as a second ranking signal on top of relevance5152Phrases that invoke this skill: "rate this skill", "quality gate", "SkillNet",53"is this skill safe to use", "why did the agent pick a bad skill".5455## When NOT to Use5657- **Routing / matching.** Which skill fits the task is `rag-mcp-tool-selection`58 or the gateway. This skill assumes a matched candidate set and filters it.59- **Libraries under ~20 curated skills.** The chapter's tip: start with a60 curated, quality-gated set (hundreds, not thousands). Below that, hand audit.61- **As a security scanner.** `safety` and `executability` are heuristic scores62 that flag a skill for review; they are not a substitute for `agent-shield` /63 SkillSpector static analysis before install.64- **As a self-generation approver.** SkillsBench shows self-generated skills65 score zero improvement; a passing composite does not license auto-install of66 agent-authored skills without human curation.6768## Process6970| Step | Input | Action | Output | Verification |71|------|-------|--------|--------|--------------|72| 1 | Skills catalog JSON (each skill carries eval_safety/completeness/executability/maintainability/cost_awareness) | `lib.load_catalog(path)` | List of skill dicts | `len(catalog) > 0`; each skill has all five `eval_*` keys in [0,1] |73| 2 | One skill dict | `lib.SkillQuality.from_skill(skill).composite` | Composite float in [0,1] | Safety+executability weighted 2x; recompute by hand for one skill to confirm /7 mean |74| 3 | Task description + catalog | `lib.retrieve_quality_gated(task, catalog, min_quality=0.6, top_k=3)` | Top-K skills passing the gate, ranked by relevance*quality | Hard gate: `eval_safety>0 AND eval_executability>0`; soft gate: `composite >= min_quality` |75| 4 | Same, varying `min_quality` | `lib.retrieve_quality_gated(..., min_quality=0.8)` | Smaller/empty result | Raising the threshold monotonically shrinks the result set |76| 5 | Catalog + a battery of task queries | `lib.monitor_gaps(catalog, queries, min_quality)` | Counts of `no_relevant_skill` vs `low_quality_filtered` events | Interpret: many no-relevant = repository too small; many low-quality-filtered = gate too permissive OR catalog quality poor |77| 6 | Selected skill | (your agent runtime) loads the SKILL.md and executes | Task result | If the top skill still misbehaves, its `eval_*` ratings are stale — re-rate |7879## Rationalizations8081| Agent rationalization | Documented rebuttal |82|------------------------|---------------------|83| "More skills always help — just retrieve top-K by relevance." | SkillsBench: self-generated (unrated) skills give zero improvement, and 2-3 focused skills per task is optimal. Relevance without a quality gate re-admits the junk that degrades retrieval at scale. |84| "Weight all five dimensions equally — it's simpler." | The chapter weights safety and executability 2x deliberately: a skill that runs unconstrained shell commands or hallucinates tool calls is dangerous regardless of maintainability. Equal weighting lets a well-documented unsafe skill outrank a plain safe one. |85| "Rank by quality alone — always pick the highest-rated skill." | The chapter ranks by `relevance * quality` multiplicatively. Pure quality ranking returns the best skill for the wrong task. Pure relevance returns the junk. The product is the point. |86| "Set min_quality once globally." | The threshold is a per-deployment risk decision: 0.4 for research to cast a wide net, 0.8+ for production healthcare. One global value ignores that risk appetite differs by system. |87| "A high composite means the skill is safe to auto-install." | Composite is a retrieval-ranking signal, not an install gate. Self-generated skills need human curation (SkillsBench) and static scanning (agent-shield/SkillSpector) before install regardless of score. |8889## Red Flags9091- **Every skill scores above min_quality.** The ratings are inflated or the gate92 is too low; the monitor will show near-zero `low_quality_filtered` events even93 on a junk-laden catalog.94- **Safety or executability is 0 but the skill still ranks.** The hard gate95 (`eval_safety>0 AND eval_executability>0`) is being bypassed — a zero on either96 dimension must exclude the skill entirely, not merely lower it.97- **Raising min_quality does not shrink results.** The soft gate is not wired;98 the threshold knob is inert.99- **High `no_relevant_skill` AND high `low_quality_filtered` together.** The100 repository is both too small and too junky — grow the curated set, do not just101 lower the gate.102- **CLI `--help` exits non-zero.** SKILL.md / CLI mismatch; the multi-harness103 invariant is broken.104105## Non-Negotiable Verification1061071. **Run the benchmark battery.**108 ```109 python cli.py benchmark110 ```111 Confirm the gate admits high-quality skills and rejects the seeded112 low-quality ones across the task battery, and that raising `--min-quality`113 monotonically shrinks the admitted set.1141152. **Hand-check one composite.**116 ```117 python cli.py rate safe-readonly-sql-query118 ```119 Recompute `(safety*2 + completeness + executability*2 + maintainability +120 cost_awareness) / 7` by hand and confirm it matches.1211223. **Prove the hard gate.**123 ```124 python cli.py retrieve "run a shell command to clean up disk" --json125 ```126 Confirm the seeded `unsafe-shell-runner` skill (eval_safety = 0) never127 appears, even when it is the most textually relevant.1281294. **JSON round-trips.**130 ```131 python cli.py retrieve "..." --json | python -c "import json,sys; json.load(sys.stdin)"132 ```133134## Security Posture135136- **Prompt injection.** Skill descriptions and `synthetic_queries` are used for137 relevance matching; treat them as untrusted if the catalog ingests138 community-contributed skills. A malicious skill could inflate its own `eval_*`139 ratings — ratings must come from an independent evaluator, not self-reported by140 the skill author. The `# TODO(production):` seam in `lib.score_relevance` and141 the rating-provenance note in `load_catalog` mark where to enforce this.142- **Data exfiltration.** No network calls in `lib.py`; catalog is read from an143 explicit path.144- **Privilege escalation.** The `safety` dimension is the flag for skills that145 execute shell / write files / hold broad credentials. A zero on `safety` hard-146 excludes; do not soften that gate to admit a convenient-but-unsafe skill.147148## Composition149150- **Composes after** `rag-mcp-tool-selection` / `mcp-gateway-two-meta-tools`:151 they return the relevant candidate set; this skill filters that set by quality.152- **Feeds** graph-based retrieval — the composite maps to the Cypher153 `relevance * quality DESC` ordering in the chapter's Example 6-3.154- **Pairs with** `agent-shield` / SkillSpector at the install boundary (this155 skill gates retrieval, those gate installation).156- **cost_awareness** links directly to the Ch8 token-economics material.157158## Source Attribution159160Distilled from *Agentic GraphRAG* (O'Reilly), Chapter 6 — Tool Orchestration,161sections "Skills: The Judgment Layer", "Skill quality evaluation", "Integrating162quality ratings into graph-based retrieval", and "Scale and retrieval". Named163references:164165- SkillsBench (DAIR.AI) — 16.2% curated-skill improvement; zero for166 self-generated; 2-3 focused skills per task optimal167- SkillNet (2026) — 200,000-skill repository; five-dimensional rating; 40%168 higher rewards / 30% fewer steps across ALFWorld, WebShop, ScienceWorld169- SkillRL — 10-20x token compression per skill; 7B model at 89.9% on ALFWorld170- Anthropic Agent Skills specification — SKILL.md progressive disclosure
AnthonyAlcaraz