Skill Eval

Skill Eval

Quick Start

# Full evaluation of all skills
uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py

# JSON output
uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py --format json

# Single skill
uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py --skill testing-tdd-london

# Only critical issues
uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py --severity critical

When to Use

• Auditing skill quality after bulk creation or migration
• Pre-release validation of the skills library
• CI/CD quality gate for skill changes
• Identifying broken cross-references after renaming skills
• Checking compliance with v2 SKILL.md template format

Core Concepts

Evaluation Dimensions

The evaluator runs 18 checks organized into three severity levels:

Critical (blocks skill usage):

• YAML frontmatter exists and parses
• Required fields present: name, description

Warning (degrades quality):

• Version follows semver, category matches directory
• Required content sections present (Quick Start, When to Use, etc.)
• Code blocks in key sections, no TODO/FIXME markers
• Cross-references in related_skills resolve to real skills

Info (improvement opportunities):

• Uses v2 template format, has optional sections (Metrics, etc.)
• No duplicate skill names across the library

Report Output

Reports include:

• Summary with pass/fail counts and percentages
• Issues grouped by severity with counts
• Per-category breakdown
• Top 10 most common issues
• Per-skill details with actionable fix suggestions

Usage Examples

Full Evaluation

uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py

Output:

================================================================
  SKILL EVALUATION REPORT
  Generated: 2026-01-29T14:30:00+00:00
================================================================

SUMMARY
----------------------------------------
  Total skills evaluated:  230
  Passed (no critical):    187  (81.3%)
  Warnings only:           102  (44.3%)
  Critical failures:        43  (18.7%)

JSON for CI/CD

uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py \
  --format json --severity critical \
  --output reports/skill-eval.json

Filter by Category

uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py --category development

Summary Only

uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py --summary-only

Audit Scripts

Run these scripts as part of skill evaluation to catch structural violations and coverage gaps:

# Check for structural violations (README presence, word count, description length, XML tags)
bash scripts/skills/audit-skill-violations.sh

# Validate skill structure (name conventions, required fields)
bash scripts/skills/validate-skills.sh

# Check which skills lack any script call reference
bash scripts/skills/skill-coverage-audit.sh

Housekeeping Issue Workflow

When asked to review the entire skill ecosystem and create a housekeeping GitHub issue, use a layered audit rather than relying on one script:

1. Run the evaluator summary for the whole ecosystem:

   uv run .Codex/skills/development/skill-eval/scripts/eval-skills.py --summary-only
   

2. Check cross-agent visibility/parity before claiming ecosystem health:

   find -L .Codex/skills -name SKILL.md -not -path '*/_archive/*' | wc -l
   find -L .codex/skills -name SKILL.md -not -path '*/_archive/*' | wc -l
   find -L .gemini/skills -name SKILL.md -not -path '*/_archive/*' | wc -l
   test -L .codex/skills && echo codex_symlink_ok || echo codex_symlink_bad
   test -L .gemini/skills && echo gemini_symlink_ok || echo gemini_symlink_bad
   

3. Add a targeted active-skill inventory that excludes _archive but deliberately reports grouping/taxonomy drift, including:
   • top-level category counts and categories with <=2 skills
   • missing category frontmatter
   • frontmatter category vs directory mismatches
   • duplicate frontmatter names
   • oversized SKILL.md files
   • missing ## Quick Start / ## When to Use
   • skills without linked scripts/, references/, templates/, or assets/
4. Search existing GitHub issues before creating a new one, and reference related open/closed skill-curation issues to avoid duplication.
5. Frame the issue as a recurring housekeeping umbrella when the user asks for periodic review: weekly deterministic report, monthly semantic/grouping review, quarterly adversarial taxonomy review.
6. Include acceptance criteria for reports, trend deltas, archive handling (_archive and _archived), category alias/waiver policy, and no issue spam.

Best Practices

• Run after creating new skills with /skill-creator to validate structure
• Use --format json in CI pipelines for machine-readable output
• Address critical issues first (missing frontmatter, invalid YAML)
• Use --severity warning to focus on actionable improvements
• Run --category filters for focused audits of specific skill areas
• If skills or index markdown changes, regenerate derived skills summary artifacts: uv run --no-project python scripts/skills/generate-skill-summary.py

Error Handling

Metrics & Success Criteria

Version History

• 1.0.0 (2026-01-29): Initial release with 18 checks, human + JSON output, category/skill filtering