# Skill Tester

> Tests Codex skill functionality with TDD approach, verifying skills work correctly through automated test scenarios and validation

- Skill: `materialofair/skill-tester` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add materialofair/skill-tester`
- Raw SKILL.md: https://api.skillmd.com/api/skills/materialofair/skill-tester/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: materialofair (https://skillmd.com/u/materialofair)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/materialofair/skill-tester

---



# Skill Tester

Test-driven development tool for Codex skills. Verifies skill functionality, validates outputs, and ensures skills work correctly before distribution.

## Capabilities

- **Functional Testing**: Tests if skills produce expected outputs for given inputs
- **Trigger Testing**: Verifies skills trigger when they should
- **Regression Testing**: Ensures changes don't break existing functionality
- **Integration Testing**: Tests skills work together correctly
- **Performance Testing**: Measures skill execution time and resource usage
- **TDD Workflow**: Write tests first, then validate skill meets requirements
- **Test Coverage Analysis**: Identifies untested skill capabilities
- **Automated Test Generation**: Creates test cases from skill examples

## TDD for Skills

### Red-Green-Refactor for Skills

**Red Phase**: Write test showing skill doesn't work yet
```
Test: "Skill should calculate P/E ratio from price and EPS"
Result: ❌ Skill doesn't have this capability
```

**Green Phase**: Implement skill to pass test
```
Test: "Skill should calculate P/E ratio from price and EPS"
Result: ✅ Skill correctly calculates 15.5 from price=$100, EPS=$6.45
```

**Refactor Phase**: Improve skill quality
```
Test: "Skill should handle edge cases (zero EPS, negative values)"
Result: ✅ All edge cases handled correctly
```

## Input Requirements

**Basic Test**:
- Skill name or path
- Test scenario (what should happen)
- Expected outcome

**Comprehensive Test Suite**:
- Skill path
- Test cases (inputs + expected outputs)
- Success criteria
- Edge cases to verify

**Example Test Case**:
```json
{
  "skill": "financial-analyzer",
  "test": "Calculate P/E ratio",
  "input": {"price": 100, "eps": 6.45},
  "expected_output": {"pe_ratio": 15.5},
  "tolerance": 0.1
}
```

## Output Formats

**Quick Test Result**:
```
Testing: financial-analyzer
Test: "Calculate P/E ratio"
Result: ✅ PASS (output: 15.5, expected: 15.5)
```

**Detailed Test Report**:
```
=== Skill Test Report ===

Skill: financial-analyzer
Tests Run: 12
Passed: 10 ✅
Failed: 2 ❌
Coverage: 83%

Failures:
1. ❌ Edge case: Zero EPS
   Input: {price: 100, eps: 0}
   Expected: Error or infinity
   Got: Division error

2. ❌ Negative values
   Input: {price: -50, eps: 5}
   Expected: Error message
   Got: Calculated -10

Recommendations:
- Add error handling for zero/negative values
- Validate inputs before calculation
- Add edge case tests to skill documentation
```

**Test Coverage Map**:
```
Capability Coverage:

✅ Calculate P/E ratio - 100% (4/4 tests pass)
✅ Calculate ROE - 100% (3/3 tests pass)
⚠️  Calculate debt ratio - 67% (2/3 tests pass, missing edge case)
❌ DCF valuation - 0% (Not implemented)

Overall: 75% coverage
```

## Test Types

### 1. Trigger Tests
**Purpose**: Verify skill triggers when it should

**Test**: "Financial-analyzer should trigger for 'calculate P/E ratio'"
```
Query: "Calculate P/E ratio for this stock"
Expected: financial-analyzer skill activates
Result: ✅ Triggered correctly
```

### 2. Functional Tests
**Purpose**: Verify correct outputs

**Test**: "Should calculate correct P/E ratio"
```
Input: price=100, eps=6.45
Expected: 15.5
Result: ✅ Output: 15.504 (within tolerance)
```

### 3. Edge Case Tests
**Purpose**: Handle unusual inputs

**Test**: "Should handle zero EPS gracefully"
```
Input: price=100, eps=0
Expected: Error message or infinity with warning
Result: ❌ Division error (needs fix)
```

### 4. Integration Tests
**Purpose**: Skills work together

**Test**: "Quality-analyzer should use skill-tester"
```
Scenario: Analyze skill quality
Expected: Skill-tester used to verify examples
Result: ✅ Integration works
```

### 5. Performance Tests
**Purpose**: Execution within acceptable time

**Test**: "Should complete analysis within 5 seconds"
```
Input: Large dataset
Expected: < 5 seconds
Result: ✅ 2.3 seconds
```

## How to Use

**Quick Test**:
```
"Test if financial-analyzer calculates P/E ratio correctly"
"Verify code-review skill works for pull requests"
"Test skill-quality-analyzer on itself"
```

**Comprehensive Test Suite**:
```
"Run full test suite on financial-analyzer with all edge cases"
"Test all capabilities of aws-solution-architect skill"
```

**TDD Workflow**:
```
"Write tests for a new skill that should calculate financial ratios"
"Create test cases for photo-enhancer before implementing it"
```

**Regression Testing**:
```
"Test if my skill changes broke anything"
"Run regression tests on code-review after updates"
```

## Test Case Structure

```yaml
test_cases:
  - name: "Basic P/E calculation"
    input:
      price: 100
      eps: 6.45
    expected_output:
      pe_ratio: 15.5
    tolerance: 0.1
    priority: HIGH

  - name: "Zero EPS edge case"
    input:
      price: 100
      eps: 0
    expected_error: "EPS cannot be zero"
    priority: CRITICAL

  - name: "Negative price edge case"
    input:
      price: -50
      eps: 5
    expected_error: "Price must be positive"
    priority: HIGH
```

## Automated Runner

Run a skill's bundled regression cases with:

```bash
node .agent/skills/local/skill-tester/scripts/run-skill-tests.js \
  --skill-path .agent/skills/local/<skill-name> \
  --mode static
```

The runner loads `<skill-path>/evals/evals.json` by default. Override it with
`--cases <path>`.

Supported case types:

- `route` checks top/allowed/forbidden skills and minimum confidence through the repository router.
- `content` checks required or forbidden text across files inside the skill directory.
- `behavior` sends a read-only prompt to an explicitly configured runner and validates its output.

Supported modes:

- `static` runs deterministic route/content cases and reports behavior cases as skipped.
- `auto` runs behavior cases only when `--runner` or `--runner-cmd` is supplied.
- `runner` requires a runner and fails if behavior replay cannot execute.

Use `--runner codex` for an ephemeral, read-only Codex replay, or
`--runner-cmd '<command>'` for a custom stdin-based runner. Never enable an
external runner implicitly.

## Test Coverage Goals

| Skill Type | Target Coverage |
|-------------|----------------|
| Critical (financial, security) | 95%+ |
| Production (widely used) | 80%+ |
| Experimental (new features) | 60%+ |
| Utility (simple tools) | 50%+ |

## Integration with Development Workflow

### Phase 1: TDD (Before Implementation)
```
1. Write test cases based on requirements
2. Document expected outputs
3. Run tests (all should fail - RED phase)
4. Implement skill
5. Run tests (should pass - GREEN phase)
6. Refactor and optimize
```

### Phase 2: Continuous Testing
```
1. Run tests after every skill change
2. Add tests for new capabilities
3. Update tests when requirements change
4. Track coverage over time
```

### Phase 3: Pre-Release
```
1. Full test suite (100% of capabilities)
2. Edge case validation
3. Performance benchmarks
4. Integration tests
5. User acceptance testing
```

## Best Practices

1. **Write Tests First**: TDD approach prevents bugs
2. **Test Edge Cases**: Zero, negative, null, empty, very large values
3. **Use Realistic Data**: Test with actual use case data
4. **Automate Testing**: Run tests automatically on changes
5. **Track Coverage**: Aim for 80%+ for production skills
6. **Document Test Results**: Keep test reports for regression checking
7. **Test Integrations**: Verify skills work together
8. **Performance Benchmarks**: Set and monitor time limits

## Common Test Patterns

### Pattern 1: Golden Master Testing
```
Input: Known input
Expected: Known correct output (golden master)
Verify: Output matches golden master exactly
```

### Pattern 2: Property-Based Testing
```
Property: "P/E ratio should always be price / EPS"
Test: Generate random valid inputs
Verify: Formula holds for all inputs
```

### Pattern 3: Snapshot Testing
```
Input: Complex data structure
Expected: Snapshot of previous correct output
Verify: Current output matches snapshot
```

## Limitations

- **Manual Test Creation**: Tests need to be written (not fully automatic)
- **No Runtime Monitoring**: Can't test skills during actual use
- **Static Testing**: Tests predefined scenarios, not all possible inputs
- **No User Testing**: Can't test user satisfaction or UX
- **Coverage Gaps**: Can't guarantee 100% coverage of all code paths

## When NOT to Use This Skill

- **Quick Experiments**: Testing adds overhead for throwaway skills
- **Debugging**: Use skill-debugger for trigger/discovery issues
- **Documentation**: Use skill-doc-generator for doc generation
- **Quality Analysis**: Use skill-quality-analyzer for static analysis

## Success Criteria

A well-tested skill should:
- ✅ Pass 100% of critical test cases
- ✅ Handle all documented edge cases
- ✅ Work within performance limits
- ✅ Integrate correctly with other skills
- ✅ Have 80%+ test coverage for production skills

## Example Test Session

```
User: "Test the financial-analyzer skill"

Codex: "Running test suite for financial-analyzer...

Test Results:
✅ Basic calculations (4/4 passed)
✅ Error handling (3/3 passed)
⚠️  Edge cases (2/3 passed)
   - Failed: Handling of extremely large numbers

✅ Integration tests (2/2 passed)
✅ Performance (under 3 seconds)

Overall: 11/12 tests passed (92%)

Recommendation: Fix large number handling before production release.
Would you like detailed failure analysis?"
```

## Related Skills

- Works with **skill-quality-analyzer**: Test quality after static analysis
- Works with **skill-debugger**: Test after fixing trigger issues
- Works with **skill-doc-generator**: Test examples in generated docs
- Complements **TDD Guard**: Enforces test-first development