Skill Evaluator

Skill Evaluator

Measure and report on skill effectiveness.

When to Use

• Validating a newly created skill
• Measuring improvement from skill adoption
• Auditing existing skills for effectiveness
• Comparing skill variants

Do NOT Use

• For one-off task execution
• When no measurable outcomes exist

Process

┌─────────────────────────────────────────────────┐
│  1. DEFINE: Set benchmark and success criteria  │
│  2. RUN: Execute skill on test cases             │
│  3. MEASURE: Collect performance metrics         │
│  4. REPORT: Summarize outcomes and trends        │
└─────────────────────────────────────────────────┘

Metrics Framework

Evaluation Types

Step-by-Step Evaluation

1. Define Benchmark

Create test cases that represent typical usage:

# Benchmark Cases

| Case | Input | Expected Output |
|------|-------|-----------------|
| Basic | Simple trigger | Minimal viable output |
| Complex | Multi-step request | Complete workflow |
| Edge | Unusual input | Graceful handling |
| Stress | Large input | Performance under load |

2. Run Skill

Execute skill on each test case:

# Record timing
START=$(date +%s)
# Execute skill...
END=$(date +%s)
DURATION=$((END - START))

3. Measure Outcomes

Collect metrics for each run:

4. Generate Report

# Skill Evaluation Report

## Summary
- Skill: <name>
- Date: <date>
- Evaluator: <agent/user>

## Results
| Metric | Value | Target | Status |
|--------|-------|--------|--------|
| Success Rate | 85% | 90% | Near |
| Avg Time | 8.3s | 10s | Pass |
| Consistency | 100% | 100% | Pass |

## Findings
- Strong: Fast execution, consistent output
- Weak: Edge case handling
- Recommendation: Add edge case handling

Evaluation Templates

Functional Test

## Functional Evaluation

### Test Cases
1. [ ] Basic trigger works
2. [ ] Multi-step process completes
3. [ ] Error handling works
4. [ ] Output format correct

### Results
Pass: X/4

Performance Test

## Performance Evaluation

### Benchmarks
| Operation | Time | Baseline | Status |
|-----------|------|----------|--------|
| Init | 0.5s | 1s | Pass |
| Execute | 2s | 5s | Pass |
| Cleanup | 0.1s | 0.5s | Pass |

Usability Test

## Usability Evaluation

### Questions
- [ ] Documentation clear?
- [ ] Steps easy to follow?
- [ ] Error messages helpful?
- [ ] Output format useful?

### Score: X/4

Comparative Evaluation

When comparing skill variants:

Automation Integration

Skills can self-evaluate:

# Self-Evaluation Checklist
- [ ] All process steps executed
- [ ] Output matches expected format
- [ ] No errors in execution
- [ ] Time within acceptable range

Reporting Frequency

Example Evaluation

User: "Evaluate the rust-development skill"

Agent: [Skill Evaluator activated]
1. DEFINE: 5 test cases (basic, complex, edge, refactor, docs)
2. RUN: Execute each with timing
3. MEASURE:
   - Success: 4/5 (80%, edge case failed)
   - Avg time: 45s per task
   - Consistency: 100%
4. REPORT:
   - Skill effective for standard work
   - Needs edge case handling improvement
   - Recommendation: Add error recovery steps

Improvement Cycle

1. Evaluate → 2. Identify gaps → 3. Update skill → 4. Re-evaluate

Track improvements over time: