Skill Eval

Skill Eval

Test a skill with parallel eval runs and iterate based on results.

Input

SKILL_PATH = $ARGUMENTS

If empty: AskUserQuestion("Which skill do you want to test? Provide the path.")

Workflow

Step 1: Validate Skill Exists

If not exists(SKILL_PATH/SKILL.md):
  ERROR "No SKILL.md found at SKILL_PATH"

SKILL_CONTENT = Read(SKILL_PATH/SKILL.md)
SKILL_NAME = extract name from frontmatter
SKILL_DESCRIPTION = extract description from frontmatter

Step 2: Load or Create Evals

EVALS_PATH = SKILL_PATH/evals/evals.json

If exists(EVALS_PATH):
  EVALS = Read(EVALS_PATH)
  Present evals to user, ask: "Use these evals, modify, or create new ones?"
Else:
  Generate 2-3 realistic test prompts based on SKILL_CONTENT
  Each prompt should be what a real user would actually say
  Include enough detail: file paths, context, specific requests
  Present to user: "Here are test cases I'd like to try. Look right, or want changes?"

Wait for user confirmation before proceeding.

Eval format:

{
  "skill_name": "SKILL_NAME",
  "evals": [
    {
      "id": 1,
      "prompt": "User's realistic task prompt",
      "expected_output": "Description of expected result",
      "expectations": [
        "The output includes X",
        "The skill used approach Y"
      ]
    }
  ]
}

Save confirmed evals to EVALS_PATH.

Step 3: Set Up Workspace

WORKSPACE = SKILL_PATH-workspace
ITERATION = find highest iteration-N in WORKSPACE + 1, or 1 if none
ITER_DIR = WORKSPACE/iteration-ITERATION

mkdir -p ITER_DIR

Step 4: Spawn All Runs

For each EVAL in EVALS, spawn TWO subagents in the SAME turn:

With-skill run:

Agent(prompt: """
Execute this task:
- Read the skill at: SKILL_PATH/SKILL.md
- Follow the skill's instructions to complete this task: EVAL.prompt
- Save all outputs to: ITER_DIR/eval-EVAL.id/with_skill/outputs/
""")

Baseline run (no skill):

Agent(prompt: """
Execute this task (no special instructions):
- Task: EVAL.prompt
- Save all outputs to: ITER_DIR/eval-EVAL.id/without_skill/outputs/
""")

Launch ALL runs (with-skill + baseline for every eval) in parallel.

Step 5: Draft Assertions While Runs Execute

While runs are in progress, draft quantitative assertions for each eval:

• Assertions must be objectively verifiable
• Give each a descriptive name
• Skip assertions for subjective outcomes (writing style, design quality)

Write eval_metadata.json for each eval directory:

{
  "eval_id": 1,
  "eval_name": "descriptive-name",
  "prompt": "The user's task prompt",
  "assertions": ["Output includes X", "File is valid JSON"]
}

Explain assertions to user while waiting.

Step 6: Grade Results

Once all runs complete:

For each run (with_skill and without_skill):

Agent(prompt: """
Read agents/skill-grader.md and follow its instructions.
- expectations: EVAL.expectations
- transcript_path: ITER_DIR/eval-EVAL.id/CONFIG/transcript.md
- outputs_dir: ITER_DIR/eval-EVAL.id/CONFIG/outputs/
- eval_prompt: EVAL.prompt
""", subagent_type: "majestic-tools:agents:skill-grader")

Step 7: Present Results

For each eval, show:

• Prompt
• With-skill pass rate vs baseline pass rate
• Key differences in outputs
• Any eval feedback from grader

Summary table:

| Eval | With Skill | Baseline | Delta |
|------|-----------|----------|-------|
| 1    | 85%       | 35%      | +50%  |

Ask user: "How do these look? Any specific feedback on the outputs?"

Step 8: Iterate (if needed)

If user has feedback:
  1. Analyze feedback — generalize, don't overfit to specific cases
  2. Improve the skill (Edit SKILL.md)
  3. Explain changes made and why
  4. Ask: "Want to rerun the evals with the updated skill?"
  5. If yes: goto Step 3 (new iteration)

If user is satisfied:
  "Skill looks good!"

Improvement Guidelines

When rewriting skills based on feedback:

• Generalize — don't overfit to the 2-3 test cases; the skill will be used many times
• Keep it lean — remove instructions that aren't pulling their weight
• Explain the why — tell the model why things matter instead of rigid MUSTs
• Look for repeated work — if all runs wrote similar helper scripts, bundle the script in the skill
• Read transcripts — check if the skill wastes time on unproductive steps

Error Handling