Skill Eval

skill-eval

Evaluating skill: $ARGUMENTS

Modes

Parse the first argument to determine mode:

Mode: eval

Test a skill against a scenario and grade the output against expectations.

Process

1. Load the skill — Read .claude/skills/<skill-name>/SKILL.md
2. Load evaluation criteria — Read the skill's ## Evaluation Criteria section (if present in SKILL.md or SKILL_TEMPLATE.md)
3. Define the scenario — Use the provided --scenario or prompt the user for one
4. Execute mentally — Walk through what Claude would do with this skill loaded, given the scenario prompt. Trace the expected decision points, tool calls, and outputs.
5. Grade against criteria — For each evaluation criterion, determine PASS/FAIL with evidence

Output

## Skill Evaluation: <skill-name>

### Scenario
> <the test scenario>

### Expected Behavior
| # | Criterion | Status | Evidence |
|---|-----------|--------|----------|
| 1 | [criterion text] | PASS/FAIL | [why] |

### Hard Gate Compliance
- [ ] All hard gates would be respected
- [ ] Red flags table would prevent shortcuts (if applicable)

### Context Budget
- Estimated SKILL.md tokens: X
- References loaded: [list]
- Total estimated tokens: X

### Verdict: PASS / FAIL / PARTIAL
[Summary of findings]

Mode: compare

Blind A/B comparison of two skill versions.

Process

1. Load both versions — Read the old and new skill files
2. Define scenario — Use provided --scenario or prompt for one
3. Evaluate independently — Run the eval process for BOTH versions, labeling them "Version A" and "Version B" (randomize which is old/new)
4. Compare blind — Without knowing which is old/new, assess which version:
   • Produces better outcomes for the scenario
   • Has stronger enforcement (hard gates, red flags)
   • Is more context-efficient
   • Handles edge cases better
5. Reveal and recommend — Reveal which was old/new, present the winner with reasoning

Output

## Skill Comparison: <skill-name>

### Scenario
> <the test scenario>

### Version A Assessment
[Strengths, weaknesses, estimated behavior]

### Version B Assessment
[Strengths, weaknesses, estimated behavior]

### Winner: Version [A/B]
**Reason:** [why this version is better for this scenario]

### Reveal
- Version A = [old/new path]
- Version B = [old/new path]

### Recommendation
[Keep new / revert to old / cherry-pick specific changes]

Mode: metrics

Analyze a skill's testability and evaluation readiness.

Process

1. Load the skill — Read SKILL.md and any references
2. Check for evaluation criteria — Does the skill have a ## Evaluation Criteria section?
3. Identify pressure scenarios — What realistic prompts would test this skill's enforcement?
4. Assess coverage — Are hard gates testable? Are red flags covered by scenarios?
5. Suggest improvements — What evaluation criteria or pressure scenarios should be added?

Output

## Skill Metrics: <skill-name>

### Evaluation Readiness
- Has evaluation criteria: YES/NO
- Has pressure scenarios: YES/NO
- Hard gates: X (testable: Y)
- Red flag entries: X

### Suggested Pressure Scenarios
1. [Scenario that would test hard gate compliance]
2. [Scenario that would trigger a red flag rationalization]
3. [Edge case scenario]

### Suggested Evaluation Criteria
- [ ] [Criterion to add to the skill]

Mode: baseline

Capture a baseline output for regression detection.

Process

1. Load the skill — Read .claude/skills/<skill-name>/SKILL.md
2. Define scenario — Use provided --scenario or prompt for one
3. Execute eval — Run the standard eval process
4. Save baseline — Write the graded output to .claude/skills/skill-eval/baselines/<skill-name>-v<version>.md

Baseline Format

# Baseline: <skill-name> v<version>

## Scenario
> <the test scenario>

## Expected Behavior
[Full eval criteria results]

## Captured: <date>

Mode: regression

Compare current skill behavior against a saved baseline.

Process

1. Load the skill — Read current .claude/skills/<skill-name>/SKILL.md
2. Load baseline — Read .claude/skills/skill-eval/baselines/<skill-name>-v<latest>.md
3. Run eval — Execute eval with the baseline's scenario
4. Diff results — Compare current eval against baseline criteria
5. Report regressions — Flag any criteria that changed from PASS to FAIL

Output

## Regression Check: <skill-name>

### Baseline: v<old> → Current: v<new>

| # | Criterion | Baseline | Current | Status |
|---|-----------|----------|---------|--------|
| 1 | [text]    | PASS     | PASS    | OK     |
| 2 | [text]    | PASS     | FAIL    | REGRESSION |

### Verdict: CLEAN / REGRESSION DETECTED
[Details of any regressions]

Rules

• This skill is for ANALYSIS only — do not modify skill files during eval or compare
• In compare mode, do NOT reveal which version is old/new until the blind assessment is complete
• Be honest about limitations — mental execution is not the same as actual execution
• Reference specific lines in skill files when citing evidence
• In baseline mode, always include the version number in the filename
• In regression mode, always compare against the most recent baseline