Skill Evaluator

Skill Evaluator

Evaluate Agent Skills against agentskills.io specification and best practices.

Core Philosophy

Good Skill = Expert-only Knowledge − What the LLM Already Knows

Before Evaluating, Ask Yourself

1. "Does this skill capture knowledge that took someone years to learn?" — If no, low D1 score.
2. "Would an expert read this and nod, or roll their eyes?" — Eye-roll = redundant content.
3. "After reading, can the LLM do something it couldn't before?" — If just faster/reminded, marginal value.

Mode Selection

Default workflow: Static first → If score >60 AND installation considered → Semi-Static → If deploying to production → Full.

Mode 1: Static Analysis

Input: SKILL.md only
Output: Gate check (Pass/Fail) + Quality Score /60

Gate Check (All Must Pass)

Any gate failure = immediate reject. Do not proceed to scoring.

Quality Dimensions (60 points)

D1: Knowledge Delta (20 pts) — MOST IMPORTANT

"If I delete this, would the LLM perform noticeably worse?"

Red flags (subtract points):

• Library tutorials ("How to use pandas")
• Generic best practices ("Write clean code")
• Definitions the LLM knows ("PDF is Portable Document Format")

Green flags (add points):

• "When X and Y, choose Z because..."
• "NEVER do X — it causes [non-obvious problem]"
• Specific numbers/thresholds ("Scale UP not down for text clarity")
• Domain-specific sequences the LLM would get wrong

D2: Mindset + Procedures (15 pts)

Look for: Diagnostic questions, priority rules, "The expert's first question is always..."

D3: Anti-Patterns (10 pts)

What counts: Specific, actionable, with surprising consequences. "NEVER use Inter font — dead giveaway of AI-generated" beats "Choose fonts carefully."

D4: Structure & Economy (15 pts)

Check for:

• {baseDir}/references/ paths with clear "when to load" instructions
• "MANDATORY if [condition]: Read..." triggers
• "Do NOT preload" markers for large references

Static report template: See {baseDir}/references/templates.md#static-report

Mode 2: Semi-Static Analysis

Input: SKILL.md + Target Environment + User Level
Output: Static score + Fit scores = /100

Step 1: Complete Static Analysis

Run Mode 1 first. If gates fail, stop.

Step 2: Environment Fit (20 pts)

Step 3: User Fit (20 pts)

Semi-static report template: See {baseDir}/references/templates.md#semi-static-report

Mode 3: Full Analysis

Input: Complete skill package + Test environment
Output: Comprehensive score /130

Step 1: Complete Semi-Static

Run Modes 1 and 2 first.

Step 2: Security Scan (Gate — Must Pass)

MANDATORY: Read {baseDir}/references/security-scan-spec.md before scanning.

Run static scan:

python {baseDir}/scripts/security_scan.py /path/to/skill
# or: node {baseDir}/scripts/security_scan.js /path/to/skill
# or: bash {baseDir}/scripts/security_scan.sh /path/to/skill

Any HIGH severity finding = instant fail. Do not proceed.

For semantic analysis (obfuscation, prompt injection, data flow): Read {baseDir}/references/security-scan-llm.md and use the LLM scan prompt.

Step 3: Trigger Testing (10 pts)

Test 5 prompt types:

Step 4: Functional Tests (20 pts)

Full report template: See {baseDir}/references/templates.md#full-report

NEVER Do (Evaluator Anti-Patterns)

1. NEVER score D1 high for tutorials — "How to use library X" is not expert knowledge, even if well-written.

2. NEVER ignore token cost — An 800-line skill that could be 200 lines is wasting 75% of context window.

3. NEVER pass security for "educational" shell=True — Legitimate purposes don't justify vulnerabilities.

4. NEVER assume environment — A skill perfect for CLI is worthless in claude.ai web.

5. NEVER conflate "comprehensive" with "good" — More content ≠ more value. Density matters.

6. NEVER skip gate checks — A skill with invalid YAML shouldn't get quality scores.

Common Failure Patterns → Fixes

Quick Reference: Scoring Cheatsheet

D1 (Knowledge Delta):     "Would deleting this make LLM worse?"
D2 (Mindset):             "Does it shape HOW to think, not just WHAT to do?"
D3 (Anti-Patterns):       "Specific NEVERs with surprising reasons?"
D4 (Structure):           "<300 lines? Progressive disclosure? Loading triggers?"
Environment:              "Will core features actually work?"
User:                     "Right audience? Right complexity level?"

The Meta-Question

After every evaluation, ask:

"Would an expert in this domain say: 'Yes, this captures knowledge that took me years to learn'?"

• Yes → The skill has value.
• No → The skill is compressing what the LLM already knows.

Bento-Ready Report Output

For visual dashboard integration, generate JSON reports where evaluation logic directly determines visual weight.

Core Principles

异常放大，正常收敛 — Information density scales with deviation severity.

Speak human, not framework — Users haven't read our evaluation docs. No D1/D2/G1 jargon.

Output Formats

Bento Report Structure

{
  "meta": { "skillName": "...", "totalScore": 48, "maxScore": 60 },
  "summary": { "verdict": "good", "oneLiner": "Ready to use with minor improvements." },
  "blocks": [
    {
      "id": "expert-knowledge",
      "type": "score",
      "importance": { "level": "normal", "reason": "Good expert content with minor redundancy" },
      "layout": { "size": "default" },
      "content": { 
        "headline": "Expert Knowledge — Good ✓",
        "detail": "Contains valuable professional insights. Some basic tutorials could be trimmed."
      }
    }
  ]
}

Importance → Layout Mapping

Visual Assets

Add visual_asset for critical/notable blocks:

• Charts: gauge (header), radar (quality overview)
• Illustrations: warning style for security issues
• Badges: verdict display

Always include fallback.text for image generation failures.

MANDATORY for bento reports: Read {baseDir}/references/bento-report-schema.md for complete JSON schema.

MANDATORY for bento generation: Read {baseDir}/references/bento-report-instruction.md for generation rules.

Converted and distributed by TomeVault — claim your Tome and manage your conversions.