Skill Check

SkillCheck — Comprehensive Skill Testing

SkillCheck audits an entire skill directory, understands every moving part, and generates targeted tests to verify the skill works as advertised. It doesn't just check "does it roughly work" — it tests every aspect: triggering, input handling, output quality, script correctness, security, best practices, and resource integrity.

How it works

Three phases: Discover → Test → Report. Discovery is always required. Testing depth depends on what the user wants. Reporting adapts to what was found.

If the user doesn't specify, default to Standard — it catches real functional problems without the time investment of Deep. If they seem in a hurry, go Quick. If they mention thoroughness, edge cases, or "test everything", go Deep.

For a worked example of a Standard Check from start to finish, see references/example-run.md.

Gotchas

• The evals.json format must match the skill-creator schema exactly (see references/eval-schema.md). Don't invent a new format — compatibility with skill-creator's benchmarking tools is the whole point.
• Mock file generation for binary formats (DOCX, XLSX, PPTX, PDF, images) requires third-party Python packages that may not be installed. Always check availability before generating and fall back to text-based equivalents if installation fails. See references/mock-files.md for the dependency list.
• Cross-reference mismatches between SKILL.md and scripts are often the root cause of multiple downstream test failures (script execution fails, eval regression fails, I/O tests fail — all for the same reason). Identify and report the root cause rather than listing each symptom as a separate unrelated failure.
• Best practice scores are advisory and reported separately — do not add them to the main pass/fail verdict percentage. A skill can be functionally perfect but structurally messy, and the report should reflect that distinction.
• Security review runs before script execution for a reason — if Critical security issues are found in scripts, warn the user before executing them at Standard or Deep depth. Don't silently run code you've just flagged as dangerous.
• Incomplete skills (TODOs, stubs, placeholder text) need a progress report, not a wall of failures. It's easy to mechanically run the standard flow and produce 15 failures that all say "this doesn't exist yet" — that's not useful. Read references/incomplete-skills.md and reframe.
• On Claude.ai without subagents, end-to-end tests run sequentially in shared context. This means earlier test results can influence later tests — an agent that just discovered a cross-reference bug might "know" to use the correct arguments on the next test. Keep this in mind when interpreting e2e results.
• The frontmatter name field must match the parent directory name per the Agent Skills spec. This is easy to miss because many platforms will still load the skill with a mismatch — but others won't, making it a silent portability bug.
• Report sections should be collapsed when empty — a Quick Check on a minimal skill should produce 5 sections, not a 12-section skeleton full of "N/A". See references/report-format.md for collapsing rules.

Phase 1: Discover

Find the skill, scan it, understand what it contains, and decide how to test it.

Find the skill

If the user gives a specific path, use it. If not, read references/finding-the-skill.md to locate the skill from conversation context or common locations.

Scan the directory

Read the entire skill directory, not just SKILL.md. Categorize every file:

Record a manifest — you'll reference this throughout testing.

Parse the SKILL.md

Extract from the frontmatter and body:

• Name and description — what it does and when it should trigger
• Declared inputs — file types, formats, schemas (pay close attention to explicitly named extensions, column names, data structures)
• Declared outputs — file types, formats, target paths, expected structures
• Step-by-step workflows — numbered or ordered procedures
• Tool/dependency requirements — libraries, CLIs, external tools
• Conditional logic — branching paths
• Constraints and rules — ALWAYS, NEVER, MUST directives
• References to bundled files — when to read other files in the skill
• Script invocations — exact commands including arguments and flags

Analyze supporting files

For each non-SKILL.md file:

• Scripts: Read the code. Identify arguments, outputs, hardcoded paths, error handling, and dependencies. Pay special attention to argument parsing — you'll compare this against how SKILL.md invokes the script.
• References: Skim for structure and purpose. Note if SKILL.md actually references them.
• Assets: Note file types and whether they're referenced. Assets will be validated for integrity during testing — see references/quick-check.md.
• Examples: Identify as input examples, output examples, or both — gold for test generation.
• Evals: Read evals.json if present. See references/eval-schema.md for the format.

Present findings

Show the user: the file manifest, what the skill claims to do, declared inputs/outputs, immediate red flags, and whether evals were found.

If the skill appears incomplete (TODOs, placeholder text, missing referenced files, stub scripts), read references/incomplete-skills.md and adjust your approach — reframe as a progress report rather than a failure report.

Then ask which depth level they'd like (if they haven't already said).

Phase 2: Test

Read the reference file for the chosen depth level. It defines exactly which tests to run, in what order, and how to evaluate results:

• Quick: references/quick-check.md — static integrity, asset validation, SKILL.md quality, cross-reference consistency, multi-skill dependencies, security review, and 22 best practice checks (references/best-practices.md)
• Standard: references/standard-check.md — everything in Quick, plus eval regression (references/eval-schema.md), script execution with mock inputs (references/mock-files.md), happy-path I/O contract validation, and workflow tests
• Deep: references/deep-check.md — everything in Standard, plus edge case inputs, end-to-end workflow simulation, triggering analysis, error recovery, and script robustness

Test plan (Standard and Deep only)

For Standard and Deep checks, present the test plan to the user before executing — they may know about domain-specific edge cases worth adding, or tests that aren't relevant. For Quick checks on simple skills, skip straight to running the tests.

Execution order

1. Static tests first — fast, free, and may reveal issues that make later tests pointless or unsafe (security problems, broken cross-references)
2. Eval regression next — the author's own tests are the most authoritative baseline
3. Script tests — run with mock inputs, verify outputs
4. I/O contract tests — generate mock files, run the skill, check results
5. End-to-end tests last — slowest and most context-heavy; earlier tests may have already uncovered the issues they'd find

For each test, record: test ID, what was tested, pass/fail, evidence, and notes. Evidence is critical — it lets the user verify your judgment and makes the report actionable.

On Claude.ai (no subagents): Run tests sequentially. Save outputs to a workspace directory so the user can inspect them after the report.

With subagents: Static tests first, then dynamic tests in parallel. Use independent subagents for end-to-end tests — shared context lets earlier tests influence later ones, masking issues that would appear in a fresh invocation.

Phase 3: Report

Read references/report-format.md for the full template and collapsing rules. Key elements:

• Summary card with pass/fail counts and verdict (HEALTHY / MOSTLY HEALTHY / NEEDS ATTENTION / CRITICAL)
• Failed test details — what was tested, what happened, why it matters, suggested fix
• Security findings — dedicated section if anything flagged, prominent banner for Critical findings
• Best practices score — separate from the main verdict
• Improvement suggestions — prioritized: Critical → Important → Nice to have

Only include report sections that have content. See the collapsing rules in references/report-format.md for which sections to skip at each depth level.