Skill Creator

Skill Creator Guide

This skill provides the authoritative standard for creating and iteratively improving Agent Skills. It combines the Anthropic Skills Standard with our local architecture rules.

Core loop: Draft skill → Write test cases → Run evals (with-skill + baseline) → Review with user → Improve → Repeat.

Your job is to figure out where the user is in this process and help them progress. Maybe they want to create a skill from scratch — help narrow intent, write a draft, create tests, run them, iterate. Maybe they already have a draft — go straight to eval/iterate. Be flexible.

Red Flags (Anti-Rationalization)

STOP and READ THIS if you are thinking:

• "I'll skip the eval step, the skill looks fine" → WRONG. Run evals — untested skills fail silently in production.
• "I can write the whole skill without talking to the user" → WRONG. Capture Intent first — assumptions cause rewrites.
• "The description is descriptive enough" → WRONG. CSO triggers are mechanical. Follow the schema.
• "This skill is too simple for a script" → WRONG. If logic > 5 lines, text instructions fail 30% of the time. Use a script.
• "I'll skip the viewer and evaluate outputs myself" → WRONG. Generate the eval viewer BEFORE evaluating — get results in front of the human ASAP.

Rationalization Table

Purpose

Enable agents to create, test, and iteratively improve Agent Skills following the Gold Standard. This skill provides both the quality standards (what makes a good skill) and the workflow engine (how to iterate to get there).

Capabilities

• Create new skills from scratch with validated structure
• Run structured evals with baseline comparison (with-skill vs without-skill)
• Grade, benchmark, and review eval results via interactive viewer
• Iteratively improve skills based on user feedback
• Optimize skill descriptions for better triggering accuracy
• Package skills into distributable .skill files
• Adapt workflow to different environments (Claude Code, Codex, Antigravity, Claude.ai, Cowork)

1. Quality Standards (Summary)

Full details: references/writing_skills_best_practices_anthropic.md Design patterns: references/skill_design_patterns.md

Skill Anatomy

Every skill follows this directory structure:

skill-name/
├── SKILL.md (Required — YAML frontmatter + markdown body)
├── examples/      # Few-shot training: input/output pairs
├── assets/        # User output: templates, files for output
├── references/    # Agent knowledge: specs, guidelines, schemas
├── scripts/       # Executable logic: Python/Bash tools
└── eval-viewer/   # (Optional) Interactive eval review UI

All subdirectories are optional. Do NOT create README.md, CHANGELOG.md, or other aux docs inside the skill folder — all instructions go in SKILL.md.

Key Rules

• Script-First: If a step requires >5 lines of if/then/else logic, it MUST be a Python script in scripts/ — agents are unreliable at executing complex logic from text.
• Inline-Block Policy: Fenced code blocks are checked two-tier — over 20 lines warns, over 60 lines fails. mermaid fences are exempt; text/console/output fences can only warn. Bulk reference material or templates SHOULD move to examples/, assets/, or references/; core procedural blocks may instead be split into labelled sub-blocks. Thresholds are config-driven (validation.quality_checks.max_inline_lines_warn / _fail).
• Graduated Language: Skills must work across LLMs (Claude, Gemini, Codex, Qwen, Llama). Use graduated instruction strength:
  • Safety-critical: MUST/ALWAYS + explain why
  • Behavioral: Explain why + imperative verb
  • Prohibited: MUST NOT + consequence
• CSO (Search Optimization): The description field determines if a skill is loaded. Start with Use when... (preferred), Guidelines for..., Helps with..., Standards for..., or Defines.... Keep under 50 words. Make descriptions "pushy" to prevent under-triggering.
  • Carry the user's own words — the verbs they type, two or three real phrasings quoted. Measured: written that way, a skill was picked up in 9 runs out of 10; described as a capability, 1 out of 10; described in a way that sounds like a refusal, 0 out of 10.
  • Read those results on the model that will run the skill. A small model picks up nothing it is not told to use by name, so descriptions cannot be compared there at all. Method: advanced-eval-patterns.md.
• Red Flags: Every skill MUST include a "Red Flags" section to prevent agent rationalization.
• Naming: Use gerund form verb-ing-noun (e.g., processing-pdfs). Always lowercase kebab-case.

Frontmatter

---
name: skill-my-capability
description: "Use when..."
tier: [0|1|2]
version: 1.0
---

Run python3 scripts/init_skill.py --help to see available Tiers. Do NOT guess tiers manually.

Required Sections in SKILL.md

1. Purpose — the "Why"
2. Red Flags — "Stop and Rethink" triggers
3. Capabilities — bulleted list
4. Execution Mode — prompt-first, script-first, or hybrid
5. Script Contract — required for script-first/hybrid (command, inputs, outputs, exit codes)
6. Safety Boundaries — explicit scope and exclusions
7. Validation Evidence — objective verification output. The verdict, not the investigation: date, what was run, counts, pass/fail. Size is bounded by quality_checks.max_validation_evidence_lines in skill_standards_default.yaml (validate_skill.py warns past it). Reasoning, per-corpus figures, refuted hypotheses and worked examples belong in references/, linked from here. A SKILL.md is loaded into context every time the skill triggers; a reference file is read only when needed.
8. Instructions — step-by-step algorithms
9. Examples — input/output pairs (see examples/SKILL_EXAMPLE_LEGACY_MIGRATOR.md)

Template: assets/SKILL_TEMPLATE.md

Execution Mode

• Mode: hybrid
• Rationale: Skill authoring needs judgement for structure/quality decisions and deterministic scripts for repeatable validation and generation.

Script Contract

• Primary Commands:
  • python3 scripts/init_skill.py <name> --tier <N> --path <ABSOLUTE-DIR> — generate a skill skeleton. Pass --path absolute: init_skill.py:143 resolves it with os.path.abspath(), i.e. against the current working directory, so the same relative path lands in a different place depending on where you invoked it from — run it from scripts/ and the skill is scaffolded outside the repository.
  • python3 scripts/validate_skill.py <skill-path> [--json] [--strict] — validate structure and compliance
  • python3 scripts/package_skill.py <skill-path> <output-dir> — package into a .skill file. Without the output directory the archive lands in the current directory.
• Inputs: skill name/path, tier, and local policy config.
• Outputs: generated skeletons, .skill archives, and a two-list verdict — errors (blocking) and warnings (reported, non-blocking). --json emits it as {skill, errors, warnings, status}.
• Failure Semantics: non-zero exit on errors. Warnings leave the exit code at 0; --strict (alias --strict-exec-policy) promotes them. This mirrors skill-enhancer/scripts/analyze_gaps.py's gaps / advisories split — in the default mode the two gates read the same config keys and return the same verdict on any one SKILL.md (WI-033), and tests/test_shared_gate_logic.py holds them there.
• --strict is per-tool, and the two do NOT agree under it. Each promotes its own advisory classes, and those differ by design: analyze_gaps.py carries prose rules ([Language]) that this gate has no counterpart for. Measured 2026-09-02: 8 of the 22 skills exit 1 under analyze_gaps.py --strict and 0 under validate_skill.py --strict. Use --strict to sweep one tool's backlog, never as a shared CI gate.

Safety Boundaries

• Operate only on explicit target skill directories.
• No broad or implicit repo-wide mutation.
• Destructive actions never default; require explicit user intent.

Validation Evidence

• validate_skill.py output and generated diffs.
• Quality checks: section coverage, CSO, inline efficiency, metadata.

2. Creating a Skill

Capture Intent

Start by understanding the user's intent. The current conversation might already contain a workflow they want to capture. If so, extract answers from conversation history first — tools used, sequence of steps, corrections made, I/O formats observed.

1. What should this skill enable the agent to do?
2. When should this skill trigger? (what user phrases/contexts)
3. What's the expected output format?
4. Should we set up test cases? Skills with objectively verifiable outputs (file transforms, data extraction, code generation) benefit from tests. Subjective skills (writing style, art) often don't. Suggest the appropriate default, let the user decide.

Interview and Research

Proactively ask about edge cases, I/O formats, example files, success criteria, dependencies. Wait to write test prompts until this is ironed out.

Check available MCPs — if useful for research, research in parallel via subagents if available, otherwise inline.

Write the SKILL.md

1. Check Duplicates: Verify in your Skill Catalog.
2. Initialize:

   python3 scripts/init_skill.py my-new-skill --tier 2
   

3. Populate: Edit the auto-generated SKILL.md. Fill in Red Flags, description, Execution Mode, Script Contract, Safety Boundaries, Validation Evidence. Consult references/skill_design_patterns.md and references/writing_skills_best_practices_anthropic.md.
4. Cleanup: Remove unused placeholder files/directories created by init.
5. Validate:

   python3 scripts/validate_skill.py ../my-new-skill
   

Test Cases

After the draft, write 2-3 realistic test prompts — things a real user would actually say. Share them with the user for confirmation, then run them.

Save to evals/evals.json. Don't write assertions yet — just prompts. You'll draft assertions while runs are in progress.

{"skill_name": "my-skill", "evals": [
  {"id": 1, "prompt": "Realistic user prompt", "files": [],
   "expectations": ["Verifiable outcome 1", "Verifiable outcome 2"]}
]}

See references/eval_schemas.md for the full schema.

3. Running and Evaluating Test Cases

This section is one continuous sequence — don't stop partway through.

Put results in <skill-name>-workspace/ as a sibling to the skill directory. Organize by iteration (iteration-1/, iteration-2/) and within that, each test case gets a directory.

Step 1: Spawn all runs (with-skill AND baseline) in the same turn

For each test case, spawn two subagents in the same turn — one with the skill, one without. Launch everything at once so it all finishes around the same time.

With-skill run:

Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/

Baseline run (depends on context):

• New skill: no skill at all → save to without_skill/outputs/
• Improving existing skill: snapshot the old version first (cp -r), point baseline at snapshot → save to old_skill/outputs/

Write eval_metadata.json for each test case with a descriptive name.

Step 2: While runs are in progress, draft assertions

Don't wait — use this time to draft quantitative assertions. Good assertions are objectively verifiable with descriptive names. Don't force assertions onto subjective outputs.

Update eval_metadata.json and evals/evals.json with assertions.

Step 3: As runs complete, capture timing data

When each subagent completes, save total_tokens and duration_ms to timing.json in the run directory. This is the only opportunity to capture this data.

Step 4: Grade, aggregate, and launch the viewer

1. Grade each run — spawn a grader subagent reading agents/grader.md. Save grading.json with fields text, passed, evidence. For programmatic assertions, write and run a script. When the skill's output is structured (JSON/numbers/files) or it shares a PASS/FAIL gate with production, prefer a deterministic script-grader that imports the production gate — see references/advanced-eval-patterns.md.

2. Aggregate into benchmark:

   python3 scripts/aggregate_benchmark.py <workspace>/iteration-N --skill-name <name>
   

   Produces benchmark.json and benchmark.md.

3. Analyst pass — read benchmark data, surface patterns (see agents/analyzer.md).

4. Launch the viewer:

   nohup python3 eval-viewer/generate_review.py \
     <workspace>/iteration-N \
     --skill-name "my-skill" \
     --benchmark <workspace>/iteration-N/benchmark.json \
     > /dev/null 2>&1 &
   VIEWER_PID=$!
   

   For iteration 2+, add --previous-workspace <workspace>/iteration-<N-1>.

5. Tell the user: "I've opened the results in your browser. 'Outputs' tab shows test cases with feedback boxes. 'Benchmark' tab shows quantitative comparison. Come back when you're done."

Step 5: Read the feedback

When the user is done, read feedback.json. Empty feedback = looks good. Focus improvements on test cases with specific complaints.

kill $VIEWER_PID 2>/dev/null

4. Improving the Skill

How to think about improvements

1. Generalize from feedback. You're iterating on a few examples to create a skill used many times. Rather than overfitty changes or oppressively constrictive MUSTs, try different metaphors or recommend different patterns.

2. Keep the prompt lean. Remove things not pulling their weight. Read transcripts — if the skill wastes time on unproductive steps, trim those instructions.

3. Explain the why. Today's LLMs are smart. When given good harness they go beyond rote instructions. If you find yourself writing ALWAYS/NEVER in all caps, reframe with reasoning. That's more powerful and effective.

4. Look for repeated work across test cases. Read transcripts — if all subagents independently wrote similar helper scripts, that script should be bundled in scripts/.

The iteration loop

1. Apply improvements
2. Rerun all test cases into iteration-<N+1>/, including baselines
3. Launch viewer with --previous-workspace
4. Wait for user review
5. Read feedback, improve, repeat

Stop when: user is happy, feedback is all empty, or no meaningful progress.

5. Description Optimization

After the skill is working well, optimize the description for better triggering accuracy.

Step 1: Generate trigger eval queries

Create ~20 eval queries — mix of should-trigger and should-not-trigger. Save as JSON:

[{"query": "realistic user prompt with details", "should_trigger": true}]

Queries must be realistic — include file paths, personal context, abbreviations, typos, casual speech. Focus on edge cases: near-misses that share keywords but need different skills.

Step 2: Review with user

Present eval set using the HTML template:

1. Read assets/eval_review.html
2. Replace __EVAL_DATA_PLACEHOLDER__, __SKILL_NAME_PLACEHOLDER__, __SKILL_DESCRIPTION_PLACEHOLDER__
3. Write to temp file and open it
4. User edits queries, exports as eval_set.json

Step 3: Run the optimization loop

Dependencies: Requires pip install anthropic and claude CLI installed. Claude Code specific — see Section 6 for other environments.

python3 scripts/run_loop.py \
  --eval-set <path-to-eval-set.json> \
  --skill-path <path-to-skill> \
  --model <model-id-powering-this-session> \
  --max-iterations 5 --verbose

This splits eval set into 60% train / 40% test, runs each query 3 times, uses Claude with extended thinking to propose improvements, iterates up to 5 times, selects by test score to avoid overfitting.

What counts as a "trigger" (VAL-2, fixed 2026-07-30). The probe registers a uuid-suffixed CLONE of the skill as a command, then watches the turn:

• a Skill call naming either the clone or the real skill counts — when the skill is already installed the model rationally invokes the canonical name, and scoring that as "not triggered" produced a 0/69 run that read as a bad description;
• a Read counts only when it loads the skill's SKILL.md or the probe's command file — not merely when the path contains the name;
• matching is EXACT after normalising plugin:skill / apps/web:deploy forms, never a substring, and Skill.args is never inspected. A call to a different skill is a non-trigger even if it quotes ours — near-miss negatives depend on this;
• the scan continues past unrelated tool calls (budget: MAX_TOOL_CALLS_SCANNED = 8) and across message_stop, because a turn is several messages and the skill call may follow a tool result.

A run reports which name matched (canonical_matches per query) and warns when the skill is installed in the probing repo: a canonical-name trigger may have been caused by the INSTALLED description rather than the candidate one under test. Zero triggers across the whole set now prints an instrument-failure warning instead of a plausible-looking summary — that ambiguity is what let VAL-2 masquerade as a description problem.

Step 4: Apply the result

Take best_description from output, update the skill's frontmatter. Show before/after and scores.

6. Environment Adaptations

Claude Code (full workflow)

All features work: subagents for parallel test runs, viewer via browser, claude -p for description optimization, package_skill.py for packaging.

Codex

Codex supports subagents. Adapt:

• Description optimization: run_loop.py uses claude -p which is Claude Code specific. Skip this step or run it manually.

Trigger evals are single-vendor today. run_eval.py spawns claude -p, writes .claude/commands/, and parses Claude Code's stream-json. Porting it to Gemini CLI, Codex, Cursor or another harness needs a per-vendor event adapter only: the detector is already split so that normalize_skill_ref / match_skill_ref / match_read_path / classify_tool_use are vendor-neutral pure functions, and just TriggerScanner.feed knows the wire format. Tests follow the same seam — tests/fixtures/fake_cli_claude_code.py replays one vendor's protocol with zero tokens and no API key, so a second vendor is a new fixture beside it rather than a rewrite. Follow the adapter precedent in .agent/skills/skill-parallel-orchestration/references/<vendor>.md.

• Viewer: If no browser, use --static <output_path> for standalone HTML.

Antigravity IDE

Full workflow available. Follow the standard process. If subagents are available, use parallel test runs. Otherwise, run tests sequentially.

Claude.ai (no subagents)

• Running tests: No subagents. Read the skill's SKILL.md, then follow its instructions yourself, one test at a time. Skip baseline runs.
• Reviewing: Skip browser viewer. Present results directly in conversation. Save output files and tell user where they are.
• Benchmarking: Skip quantitative benchmarking.
• Description optimization: Requires claude CLI. Skip on Claude.ai.
• Packaging: package_skill.py works anywhere with Python.

Cowork (headless)

• Subagents work. If timeouts occur, run tests sequentially.
• No browser — use --static <output_path> for viewer. User clicks link to open HTML.
• Feedback: "Submit All Reviews" downloads feedback.json as file.
• GENERATE THE EVAL VIEWER BEFORE evaluating outputs yourself — get results in front of the human ASAP.

7. Advanced: Blind Comparison

For rigorous A/B comparison between skill versions, read agents/comparator.md and agents/analyzer.md. Give two outputs to an independent agent without telling it which is which, let it judge quality.

Optional. Most users won't need it — the human review loop is usually sufficient.

8. Package and Present

If the present_files tool is available:

python3 scripts/package_skill.py <path/to/skill-folder>

Direct the user to the resulting .skill file.

9. Scripts & Tools Reference

scripts/ — Core automation

• init_skill.py: Generate compliant skill skeleton
• validate_skill.py: Enforce structure, frontmatter, CSO, execution-policy
• skill_utils.py: Config loader (defaults + project overlay) + parse_skill_md()
• aggregate_benchmark.py: Compute benchmark summary from grading.json files. --bootstrap adds a seeded confidence interval on the pass-rate delta (multi-rep + interval).
• verify_pin.py: Re-aggregate committed run results and assert they match a committed benchmark.json — the generic "no silent metric drift" pin for CI (see references/advanced-eval-patterns.md §3; full tutorial in-repo at docs/Manuals/skill-evals_guide.md, not bundled in the standalone skill).
• generate_report.py: Build static HTML report from benchmark.json
• run_eval.py: Run trigger evaluation queries via CLI
• run_loop.py: Main eval + improvement loop for description optimization
• improve_description.py: Improve description using Claude with extended thinking
• package_skill.py: Package skill into .skill (ZIP) file

eval-viewer/ — Interactive eval review

• generate_review.py: Launch interactive eval viewer (serves viewer.html)
• viewer.html: Benchmark viewer with Outputs + Benchmark tabs

10. Resources Reference

• references/writing_skills_best_practices_anthropic.md — Full authoring guide (Anthropic standard)
• references/skill_design_patterns.md — Degrees of Freedom, Progressive Disclosure, EDD
• references/default_parameters.md — Config defaults and resolution order
• references/eval_schemas.md — JSON schemas for evals, grading, comparison, analysis
• references/output-patterns.md — Templates for agent output formats
• references/workflows.md — Designing skill-internal workflows
• references/persuasion-principles.md — Psychological principles for instructions
• references/testing-skills-with-subagents.md — TDD methodology for skills
• references/advanced-eval-patterns.md — Production-grade eval patterns: deterministic script-grader, pinning, set diversification ("mirage"), A/B isolation, multi-rep. Full tutorial: docs/Manuals/skill-evals_guide.md
• agents/grader.md — Evaluate assertions against outputs
• agents/comparator.md — Blind A/B comparison
• agents/analyzer.md — Post-hoc analysis of results