Skill Evaluator

Skill Evaluator

Score any Claude Code skill against a rubric reverse-engineered from the most-starred skill repositories on GitHub.

Tradeoff: This rubric rewards general-purpose, widely distributable skills. Domain-specific or internal-tooling skills will score lower on portability — that's by design, not a flaw.

How to Run

1. Determine the input. Check for:

   • A skill name the user mentions (e.g. "evaluate my karpathy-guidelines skill") → locate the file at ~/.claude/skills/{name}/SKILL.md and Read it
   • A file path in the user's message → Read the file
   • A GitHub URL → Fetch the raw content
   • Pasted SKILL.md content directly in the chat → Use as-is
   • No input provided → Ask: "Please share the skill name, a file path, a GitHub URL, or paste the skill content directly." Do not proceed until input is received.

2. Read the full skill content. If the file does not exist or the URL is unreachable, tell the user and stop.

3. Determine the evaluation scope:

   • Single skill — a standalone SKILL.md file. Evaluate only what is in that file and its immediate sibling files (e.g. references/, scripts/).
   • Plugin / repo collection — a GitHub repository containing multiple skills, agents, commands, or hooks (e.g. ECC, superpowers). Evaluate the full repository: individual skills are representative samples, but also include repo-level evidence such as CI pipelines, test suites, install scripts (install.sh, install.ps1), and repo-level docs (SECURITY.md, CODE_OF_CONDUCT.md, CHANGELOG.md).

   State the scope at the top of the output: Scope: Single skill or Scope: Plugin collection.

   Critical: for a plugin collection, do NOT sample only one or two SKILL.md files and ignore the rest of the repo. Missing repo-level infrastructure (tests, CI, install scripts) will produce artificially low scores on Verifiability, Portability, and Maintenance Maturity.

4. Classify the skill category before scoring:

   • Tool skill — produces structured output, processes files, calls APIs, or automates a specific task (examples: PDF processor, code reviewer, UI generator, data query tool)
   • Guideline skill — modifies how the model behaves or thinks; no structured output; composed of rules, principles, or style guidance (examples: karpathy-guidelines, commit style guides, tone policies)
   • If unclear, default to Tool skill.

   State the category at the top of the output: Category: Tool skill or Category: Guideline skill.

5. Score each dimension using the rubric below. Apply the correct weight column for the category.

6. Output the scorecard in the format below.

7. For every dimension scoring below 70% of its maximum, give one specific, actionable improvement with a concrete example.

8. Optional — Behavioral eval note: If evals/fixtures/<skill-name>/ exists in the repository, note at the end of the scorecard:

   Behavioral eval available: python evals/run.py --mode behavioral This is a signal that the skill can be tested for actual behavioral impact, not just document quality. If it does not exist, suggest adding it as the highest-value improvement for Verifiability.

Output Format

Always output in this exact structure:

Fill in <Max> from the Category weights table for the category detected in step 3.

## Skill Evaluation: {skill name}

Scope: {Single skill / Plugin collection}
Category: {Tool skill / Guideline skill}

| Dimension | Score | Max | Notes |
|-----------|-------|-----|-------|
| Trigger Clarity | X | 20 | one-line observation |
| Instruction Specificity | X | <Max> | one-line observation |
| Reference Density | X | <Max> | one-line observation |
| Verifiability | X | <Max> | one-line observation |
| Tradeoff Transparency | X | <Max> | one-line observation |
| Portability | X | <Max> | one-line observation |
| Maintenance Maturity | X | <Max> | one-line observation |
| **Total** | **X** | **100** | |

### Tier: {Bronze / Silver / Gold / Elite}
{One sentence summary of the skill's overall quality.}

### Top 3 Improvements
1. **{Dimension}**: {Specific change with example if possible}
2. **{Dimension}**: {Specific change with example if possible}
3. **{Dimension}**: {Specific change with example if possible}

Tier thresholds:

• Elite: 85–100
• Gold: 70–84
• Silver: 50–69
• Bronze: 0–49

Scoring Rules

• Score based on what is present in the skill, not what it could theoretically do.
• Do not infer intent — if something is not written, it does not exist.
• Be strict. A karpathy-guidelines-quality skill (176k★) scores ~78/100. Calibrate accordingly.
• If the skill is a CLAUDE.md or command file (not a SKILL.md), note this and still score what you can.

Rubric

Derived from analysis of the 9 most-starred Claude Code skill repositories (June 2026).

Source repos: obra/superpowers (229k★), affaan-m/ECC (216k★), multica-ai/andrej-karpathy-skills (176k★), anthropics/skills (151k★), mattpocock/skills (131k★), nextlevelbuilder/ui-ux-pro-max-skill (92k★), thedotmack/claude-mem (83k★), JuliusBrussee/caveman (73k★), OthmanAdi/planning-with-files (23k★)

Category weights

Dimension 1 — Trigger Clarity (20 pts)

What it measures: Does the description field tell the model when to invoke this skill, not just what it does?

Examples:

• Elite: "Use when the user asks to evaluate, score, review, rate, or improve a skill — or when they share a SKILL.md and want feedback" ✓
• Poor: "A skill for reviewing skills" ✗

Dimension 2 — Instruction Specificity (Tool: 15 pts · Guideline: 24 pts)

What it measures: Does the body give the model a concrete decision tree or procedure to follow, not just a description of what output to produce?

What to look for: Numbered steps, conditional logic, explicit fallback behavior.

Dimension 3 — Reference Density (Tool: 15 pts · Guideline: 8 pts)

What it measures: Is there bundled reference material (data, examples, templates, scripts) that the model can use directly — not just instructions?

What to look for: references/ folder, scripts/ folder, embedded data tables, code snippets with real values.

Dimension 4 — Verifiability (Tool: 15 pts · Guideline: 5 pts)

What it measures: Can success be verified? Does the skill define what "done" looks like, or better yet, include evals/benchmarks that test actual behavioral change?

There are two levels of verification — score at whichever level is present:

What to look for: evals/fixtures/<skill>/tasks.json (behavioral), evals/ folder with prompts (static), in-content success criteria (guideline skills), defined output templates, CI-committed snapshots.

Static vs. Behavioral distinction: Static evals check whether the evaluator scores consistently. Behavioral evals check whether the skill itself changes model outputs. A skill with only static evals — or a guideline skill that defines verifiable success criteria in its instructions — scores 9–12 (Tool) / 3–4 (Guideline); committed behavioral fixtures are required for the top band.

Dimension 5 — Tradeoff Transparency (Tool: 10 pts · Guideline: 18 pts)

What it measures: Does the skill honestly state its limitations, biases, or when NOT to use it?

Example of elite pattern (karpathy-guidelines):

Tradeoff: These guidelines bias toward caution over speed. For trivial tasks, use judgment.

Dimension 6 — Portability (Tool: 15 pts · Guideline: 18 pts)

What it measures: Can anyone install and use this without project-specific context? Does it work across multiple agent harnesses?

Dimension 7 — Maintenance Maturity (Tool: 10 pts · Guideline: 7 pts)

What it measures: Does the repository show signs of being maintained and trustworthy over time?

Calibration Reference

Scores use Tool skill weights unless the Category column shows "Guideline".