Evaluate Plugin Or Skill

Evaluate Plugin or Skill

A plugin is ready to share when its structure is correct, its activation is reliable, its content is disciplined, its documentation answers the three install-time questions, and its anti-patterns are absent. This workflow checks all five dimensions against the same standards skill-foundry and plugin-dev enforce — and returns a single verdict instead of a wall of unsorted feedback.

When to use this workflow

• You finished a plugin or skill and want to know if it is shippable
• You are reviewing a third-party plugin before installing or recommending it
• You drafted a SKILL.md and want to know whether the description will actually trigger
• You want a structured second opinion before publishing to a marketplace
• You need feedback that is ranked by priority — not a flat list of nitpicks

When NOT to use this workflow

• Building a new plugin from scratch → use build-a-plugin
• Extending or fixing an existing plugin → use update-a-plugin
• Authoring a single new SKILL.md → use write-your-own-skill
• Measuring runtime agent quality (eval suites for production agents) → use evaluate-and-improve-agent
• Fixing a single broken validation check → run plugin-validation directly

Prerequisites

Install the underlying plugins so the audit has real depth:

/plugin install plugin-validation@skillstack
/plugin install plugin-evaluation@skillstack
/plugin install skill-foundry@skillstack
/plugin install plugin-documenter@skillstack

Without these installed, the workflow still runs as a checklist — but the depth and the live eval harness disappear.

Core principle

A verdict beats a list. Ten unranked findings overwhelm; one verdict with three ranked actions ships. The audit sorts every finding by severity (blocker, gap, polish) and folds them into one of three outcomes: SHIP, IMPROVE, or REWORK. Authors get a clear next step instead of a homework pile.

Secondary principle: measure activation, do not assume it. Most plugins fail in the wild because their description does not trigger reliably — not because their content is bad. Trigger evals are non-negotiable; if none exist, this workflow generates seed evals and runs them before issuing a verdict.

Third principle: eat the dog food. This skill audits other skills against skill-foundry's anti-patterns. It must itself avoid those anti-patterns — lean SKILL.md, references on demand, decision tables over prose.

The phases

Phase 1 — Intake and classification

Detect what the user provided. Inputs come in three forms: a GitHub URL, a local plugin directory, or a single SKILL.md path.

Classify the artifact:

• Plugin — has .claude-plugin/plugin.json and a skills/ (or hooks/, mcp-servers/, agents/, commands/) directory
• Single skill — has a single SKILL.md with frontmatter, no plugin manifest
• Ambiguous — neither — ask the user before proceeding

Inventory components without deep-reading: count skills, hooks, MCP servers, subagents, commands. Note size of each SKILL.md in lines. This inventory drives which phases are relevant.

Output: a one-paragraph summary of what is being evaluated and which phases will run.

Phase 2 — Structural validation (plugin-validation)

Load the plugin-validation skill.

Check, for plugins:

• plugin.json has name, version, description, author, license
• Each skills/<name>/SKILL.md exists and has YAML frontmatter
• name in frontmatter equals the directory name
• All referenced files (references/*.md, scripts/*.py, etc.) exist on disk
• Plugin name and version follow conventions of the target marketplace
• No SKILL.md exceeds 500 lines (skill-foundry hard limit)

For single skills, run a subset: frontmatter validity, name field present, referenced files exist.

If plugin-dev/scripts/validate_plugins.py is available, run it for objective output. Otherwise, run the checklist by hand.

Severity classification:

• Blocker — manifest missing, name mismatch, frontmatter invalid, broken reference
• Gap — missing optional but expected fields (keywords, repository URL)
• Polish — convention drift (kebab-case violations, unusual naming)

Output: structural score (blocker count, gap count, polish count).

Phase 3 — Skill content audit (skill-foundry)

Load the skill-foundry skill.

For each SKILL.md, audit against the five canonical anti-patterns:

Also check the structural quality bar:

• Description has trigger phrases AND a NOT clause (target: ≥3 trigger keywords, ≥2 NOT cases)
• SKILL.md ≤500 lines (≤200 preferred for activation skills)
• Decision tables, not paragraphs, for routing logic
• Progressive disclosure: heavy content lives in references/, not inline

Severity classification:

• Blocker — Everything Skill, Reference Illusion, description with no trigger keywords
• Gap — Description Soup, no NOT clause, missing decision tables
• Polish — Template Theater, mild Orphaned Sections

Output: per-skill anti-pattern findings with severity.

Phase 4 — Activation evaluation (plugin-evaluation)

Load the plugin-evaluation skill.

If skills/<skill>/evals/trigger-evals.json exists, run the harness:

python3 plugin-dev/scripts/run_eval.py \
  --plugin-dir <path> --skill <name> --mode trigger

If no trigger evals exist, this is itself a finding — but do not stop. Generate seed evals: 5 positive cases (varied phrasings of when the skill should activate) and 5 near-miss negatives (queries that look related but should NOT trigger). Run those.

Targets:

• Positive trigger rate ≥ 90%
• Negative no-trigger rate ≥ 95%

Severity classification:

• Blocker — positive trigger rate < 60%, or negative no-trigger rate < 80% (skill collides with other skills)
• Gap — positive 60–89%, or negative 80–94%
• Polish — positive 90–94%, or negative 95–98%
• No finding — positive ≥ 95% AND negative ≥ 99%

Output: trigger-rate table per skill.

Phase 5 — Output evaluation (plugin-evaluation, optional)

If skills/<skill>/evals/evals.json exists, run output evals to check whether the skill produces correct results when activated. Otherwise, mark as a gap and proceed — output evals are nice-to-have for shipping, mandatory for production-critical plugins.

Severity classification:

• Blocker — output pass rate < 70% on existing evals
• Gap — no output evals exist for a plugin marketed as production-ready
• Polish — output pass rate 70–89%

Output: output-eval pass rate (or absence note).

Phase 6 — Documentation and shareability

The best plugin documentation answers three questions in install order: what problem does this solve, how do I install it, what can I do with it. Check whether the README answers these in that order.

Checklist:

• README opens with a problem statement, not "this plugin provides…"
• Installation block exists and works as written
• At least three concrete example prompts, scenarios, or invocations
• License and attribution footer present
• CHANGELOG.md exists with at least one version entry
• Plugin manifest keywords are searchable (matches actual use cases, not generic words like "tools" or "ai")

Severity classification:

• Blocker — no README, broken install instructions, no license
• Gap — README leads with capabilities instead of the problem; no example prompts; no CHANGELOG
• Polish — keyword stuffing, missing screenshots for visual plugins

Output: documentation gap list.

Phase 7 — Verdict synthesis

Score each dimension 0–3:

Aggregate the five scores (structure, content, activation, output, documentation) into one verdict:

• SHIP — every dimension ≥ 2 AND total ≥ 12. Worth sharing as-is or with cosmetic polish.
• IMPROVE — one or two dimensions at 1, none at 0. Concrete, addressable fixes. List the top three ranked actions; the author can ship after addressing them.
• REWORK — any dimension at 0, OR three+ dimensions at 1, OR positive trigger rate < 60%. The plugin needs a structural revisit; suggest restarting from build-a-plugin (for plugins) or write-your-own-skill (for single skills).

Output: a one-screen verdict report with:

1. Verdict (SHIP / IMPROVE / REWORK) — single word, top of the report
2. Dimension scores table (5 rows, score + one-sentence reason each)
3. Top three ranked actions (only the ones that move the verdict — skip polish-only items unless the verdict is already SHIP)
4. If REWORK: which workflow to restart from and why

Gates and failure modes

The most common failure mode is issuing SHIP without measuring activation. If you cannot point to a trigger pass rate, you have not evaluated the plugin — you have read it.

Output artifacts

• A verdict report (SHIP / IMPROVE / REWORK) with dimension scores and ranked actions
• Optional: generated trigger-evals.json seed file if none existed
• Optional: a one-page diff between the plugin's current state and shippable state, suitable for handing back to the author

Related workflows and skills

• build-a-plugin — when REWORK is the verdict for a plugin, restart here
• write-your-own-skill — when REWORK is the verdict for a single skill
• update-a-plugin — when IMPROVE is the verdict and the author wants a guided fix loop
• plugin-validation, plugin-evaluation, skill-foundry, plugin-documenter — the underlying skills this workflow composes

Workflow part of skillstack-workflows by Viktor Bezdek — licensed under MIT.