# Create Evals

> Generate diverse evaluation scenarios that test whether an agent using a skill produces correct output, including happy-path cases, edge cases, and failure scenarios across varying complexity levels. Produces structured scenario directories containing task prompts, weighted scoring checklists, and optional input fixtures for each scenario. Use when the user says "create evals", "generate eval scenarios", "write evals", "test this skill", or before publishing a skill.

- Skill: `tovrleaf/create-evals` (Agent Skill, multi-file: 20 files)
- Install (CLI): `npx skillmds@latest add tovrleaf/create-evals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tovrleaf/create-evals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: tovrleaf (https://skillmd.com/u/tovrleaf)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tovrleaf/create-evals

---


# Create Evals

Generate diverse eval scenarios for a skill. Each scenario
tests whether an agent following the skill produces the
expected output given a realistic task.

## Workflow

1. **Identify target** — Determine which skill to create
   evals for from the user's request.

2. **Read the skill** — Read the skill's SKILL.md and any
   `references/` to understand what behaviors it enforces.

3. **Plan scenarios** — Design 3–5 diverse scenarios:
   - At least one straightforward happy-path case
   - At least one edge case or tricky situation
   - Vary complexity (simple, moderate, challenging)
   - Cover different aspects of the skill's workflow

4. **Create scenario directories** — One directory per
   scenario in the skill's `evals/` directory. Use
   descriptive kebab-case names:
   ```text
   skill-name/
   └── evals/
       ├── scenario-happy-path/
       ├── scenario-edge-case/
       └── scenario-complex/
   ```

5. **Write files** — Each scenario directory contains:
   - `scenario.json` — Metadata (see structure below)
   - `criteria.json` — Weighted scoring checklist
   - `task.md` — Realistic task prompt
   - `inputs/` — Optional fixture files

6. **Verify** — Confirm all criteria weights sum to exactly
   100 per scenario. Run the validation script:
   ```bash
   ./scripts/check-eval-weights.sh skills/<name>
   ```

7. **Report** — List the scenarios created with descriptions.

## File Formats

### scenario.json

```json
{
  "description": "Brief description of the scenario"
}
```

Add `"include": ["./inputs"]` when input fixtures exist.

For conversational skills (no tool use), add:
```json
{
  "description": "Brief description",
  "sandbox": false
}
```

Default is `"sandbox": true` (requires Docker). Set to
`false` for skills that only produce text output.

### criteria.json

```json
{
  "context": "What the scenario tests",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "criterion-name",
      "description": "Observable behavior to verify",
      "max_score": 15
    }
  ]
}
```

### task.md

```markdown
## Problem/Feature Description

Realistic scenario description.

## Output Specification

What files or artifacts the agent must produce.
```

## Criteria Rules

- Weights MUST sum to exactly 100 per scenario
- Weight guide:
  - 12–15: Core skill behaviors
  - 10–12: Important explicitly-emphasized rules
  - 6–8: Supporting rules that matter but aren't the focus
  - 4–6: Minor style or formatting details
- Each description must be specific enough for binary
  pass/fail scoring — no subjective criteria

## Task Rules

- Two sections: Problem/Feature Description and Output
  Specification
- Self-contained and realistic — a believable scenario
- Never reference the skill being tested
- Keep focused on the skill's domain

## Input Files

- Optional — only include when the scenario needs existing
  project context
- Keep minimal — just enough to make the scenario concrete
- Must be realistic (real-looking code, configs, docs)

## Example Scenario

For a commit-conventions skill:

```text
evals/
└── scenario-breaking-change/
    ├── scenario.json
    ├── criteria.json
    ├── task.md
    └── inputs/
        └── diff.patch
```

Where `criteria.json` might weight "includes breaking
change footer" at 15, "imperative mood header" at 12,
"scope present" at 10, etc., summing to 100.

## Boundaries

- DOES generate eval scenario files
- Does NOT run evals or judge output
- Does NOT modify the skill itself
- Does NOT publish or deploy

## Common Failures

- **Weights not summing to 100** — always verify arithmetic
  before writing criteria.json.
- **Criteria too vague** — "well-written commit message" is
  unjudgeable. "Header uses imperative mood" is binary.
- **Task leaks skill name** — the task must not hint at
  which skill is active or what behaviors are expected.
- **Scenarios all test the same aspect** — cover different
  parts of the workflow, not variations of one case.
- **Unrealistic inputs** — fixture files must look like
  real project artifacts, not toy examples.