Skill Eval Scaffolder

Skill Eval Scaffolder

Create minimal, schema-valid eval manifests for repository skills.

Scope: Eval scaffolding only. Do not run live behavioral evals unless the maintainer explicitly approves.

Dispatch

Workflow

1. Confirm skills/<name>/SKILL.md exists.
2. Run the bundled scaffold script:

uv run python skills/skill-eval-scaffolder/scripts/scaffold_evals.py <name>

3. Add explicit, implicit, and negative-control cases before claiming adequate coverage.
4. Validate with the bundled check script.

Manifest Shape

Each evals/evals.json uses:

• skill_name — kebab-case directory name
• evals — non-empty list of cases with id, prompt, expected_output, optional files, assertions

Validation Contract

uv run python skills/skill-eval-scaffolder/scripts/check.py

Critical Rules

1. Never overwrite an existing eval manifest without explicit user approval.
2. Include at least one explicit-invocation case in new scaffolds.
3. Run validate_evals.py before declaring the scaffold complete.
4. Route live eval execution to skill-creator eval mode or maintainer CI workflows.
5. Keep prompts grounded in the skill dispatch table and NOT-for boundaries.