Quality Criteria

Quality Criteria

Measurable criteria for evaluating the AI-facing skill output produced by user-docs-to-ai-skill. Apply this checklist in Phase 5 before declaring done. Every failing item must be fixed before reporting completion.

Table of Contents

1. Checklist — Apply Before Done
2. Scoring Guide
3. Common Failure Modes
4. Distinguishing Good from Mediocre

Checklist categories: Frontmatter — SKILL.md Structure — Reference Files — Links — Coverage — Workflow Identification

Checklist — Apply Before Done

Run all items. Mark each PASS or FAIL. Fix all FAILs before proceeding.

Frontmatter

• FM-1: name field matches the skill directory name exactly
• FM-2: description is a single line — no >-, |-, >, | multiline indicators
• FM-3: description contains no colons except in URLs
• FM-4: description front-loads a trigger scenario ("Use when...")
• FM-5: description names the domain tool or library explicitly
• FM-6: allowed-tools is present and uses comma-separated string format

SKILL.md Structure

• SK-1: SKILL.md contains a Mermaid workflow flowchart covering primary task types
• SK-2: Every reference file in references/ is linked from SKILL.md
• SK-3: SKILL.md does not duplicate content that belongs in a reference file
• SK-4: SKILL.md sections describe what to find in each reference file (2-3 sentences each)
• SK-5: SKILL.md token count is under 4000 — estimate by character count (1 token ≈ 4 chars)

Reference Files

• RF-1: At least one reference file exists
• RF-2: All reference files have a # title heading
• RF-3: Reference files longer than 50 lines have a Table of Contents
• RF-4: All code blocks have language specifiers (bash, toml, python, text, etc.)
• RF-5: Verbatim content (CLI flags, param names, error messages) is preserved exactly — not paraphrased
• RF-6: No reference file contains motivational prose, analogies, or narrative scaffolding
• RF-7: Decision logic uses Mermaid flowcharts — not prose descriptions of conditions

Links

• LK-1: All links from SKILL.md to references use [text](./references/filename.md) syntax
• LK-2: All referenced files exist at the linked paths
• LK-3: No absolute paths used in any link

Coverage

• CV-1: Every major section of the source docs is represented by at least one atom in a reference file
• CV-2: CLI commands or API methods from source docs are in the output — none silently dropped
• CV-3: Error conditions documented in source docs are captured in a reference file
• CV-4: Configuration parameters from source docs appear with their types and defaults

Workflow Identification (Phase 1.5)

• WF-1: Workflow-shaped content identified and delegated to process-siren
• WF-2: resources/workflows/ directory contains validated Mermaid files for all identified workflows
• WF-3: Every workflow file is linked from SKILL.md under a ## Workflows section
• WF-4: No workflow-shaped prose remains as bullet lists in reference files

AI-Facing Language

• AI-1: Instructions use imperative form ("Extract all parameter names") not descriptive ("The skill will extract...")
• AI-2: No marketing copy, no motivational framing, no "this will help you" sentences
• AI-3: Conditional logic is expressed in structured form (flowchart or explicit if/then) — not in prose

Scoring Guide

Count FAILs after first pass:

flowchart TD
    Count([Count FAIL items]) --> Q1{FAIL count?}
    Q1 -->|0| Done([Output is publication-ready])
    Q1 -->|1-3| Minor[Fix all items\nRe-run checklist\nExpect PASS on second run]
    Q1 -->|4-8| Moderate[Identify root cause\nOften one structural issue causes multiple FAILs\nFix root cause first\nThen re-run checklist]
    Q1 -->|9+| Rebuild[Output has fundamental structural problem\nDo not patch — re-extract from source docs\nIdentify which Phase produced the errors]

Common Failure Modes

Failure Mode 1 — Prose Dump

Symptom: Reference files contain long paragraphs copied from the source docs.

Detection: Any paragraph longer than 4 sentences in a reference file.

Root cause: Extraction Phase skipped or incomplete — raw doc content pasted instead of atoms extracted.

Fix: Re-run Phase 1 extraction on the affected sections. Extract atoms, then rewrite the reference section using those atoms.

Failure Mode 2 — Missing Frontmatter

Symptom: SKILL.md frontmatter block is absent or incomplete.

Detection: The file opens without ---, or name/description/allowed-tools are missing.

Root cause: Phase 4 (Write SKILL.md) executed before reading skill-structure-guide.md frontmatter rules.

Fix: Add correct frontmatter. Validate each field against FM-1 through FM-6 checklist items.

Failure Mode 3 — No Progressive Disclosure

Symptom: SKILL.md contains paragraphs of domain knowledge instead of links to reference files.

Detection: Any section of SKILL.md longer than 10 lines that is not a flowchart, a quick-reference block, or a reference index.

Root cause: Author treated SKILL.md as the knowledge file rather than the routing layer.

Fix: Move inline knowledge blocks into appropriately-named reference files. Replace with 2-3 sentence description + link.

Failure Mode 4 — Paraphrased Identifiers

Symptom: CLI flags, parameter names, or error messages are reworded from the source.

Detection: Compare a sample of 5 atoms to source docs. Any deviation in flag names, param names, or error strings is a failure.

Example:

# Source doc: --python-version 3.12
# Paraphrased (WRONG): python version flag: 3.12
# Verbatim (CORRECT): --python-version 3.12

Root cause: Extraction Phase applied abstraction rules to identifiers (should only abstract narrative prose).

Fix: Re-extract the affected sections using verbatim preservation rules from extraction-patterns.md.

Failure Mode 5 — Empty Themes

Symptom: A reference file exists but contains fewer than 3 atoms of knowledge.

Detection: Reference file body under 20 lines (excluding header and ToC).

Root cause: Over-splitting during Phase 2 thematic grouping.

Fix: Merge the thin reference file into an adjacent theme. Update the link in SKILL.md.

Failure Mode 6 — Broken Links

Symptom: SKILL.md links to reference files that don't exist or use wrong paths.

Detection: Read each linked path with Read(file_path). Any 404-equivalent is a failure.

Fix: Verify every link after writing all files. Correct any path that doesn't resolve.

Failure Mode 7 — Vague Description

Symptom: The description field doesn't trigger auto-invocation for the intended use cases.

Detection: Read the description and ask: "Would Claude load this skill when a user says 'help me configure ty.toml'?" If the answer is uncertain, it's vague.

Example:

# Vague (WRONG)
description: Knowledge about ty tool and its features

# Triggerable (CORRECT)
description: Use when working with ty — running type checks, configuring ty.toml, interpreting error codes, or targeting specific Python versions

Fix: Rewrite description using the template in skill-structure-guide.md. Include explicit trigger scenarios.

Distinguishing Good from Mediocre

A mediocre output skill:

• Has a description that doesn't trigger on real user questions
• Contains reference files that are reformatted copies of the source docs
• Uses SKILL.md as a long-form knowledge document
• Has no decision logic — only flat prose
• Paraphrases CLI flags and parameter names
• Has no workflow flowchart

A good output skill:

• Triggers auto-invocation on natural user questions about the domain
• Extracts atoms from source docs — facts and constraints, not narrative
• Uses SKILL.md only for routing and linking
• Represents conditional logic as Mermaid flowcharts
• Preserves verbatim all identifiers (flags, params, errors, enum values)
• Has a clear workflow in the SKILL.md body that maps task types to reference files

The fastmcp-creator skill in plugins/fastmcp-creator/skills/fastmcp-creator/ is the reference exemplar. Compare the output against it before declaring done.