Evaluate Improve

/evaluate:improve

Analyze evaluation results and suggest concrete improvements to a skill.

The apply-path machinery is split into references/ by the flag that needs it — the delta-verify gate and --best-of ranking are read only when you are actually applying an edit.

When to Use This Skill

Parameters

Parse these from $ARGUMENTS:

Execution

Step 1: Load eval results

Read the most recent benchmark from:

<plugin-name>/skills/<skill-name>/eval-results/benchmark.json

If no results exist, suggest running /evaluate:skill first and stop.

Also read the current SKILL.md to understand the skill.

Capture the source-failure set. From the benchmark, record the set of eval-case IDs that failed with the skill active — these are the cases the forthcoming edit is meant to fix, and they are the input to the delta-verify gate below:

cat <plugin>/skills/<skill>/eval-results/benchmark.json \
  | jq -r '[.cases[] | select(.with_skill.passed == false) | .id]'

This set is distinct from the golden evals.json suite as a whole: the golden set measures overall pass rate, the source-failure set measures whether the edit fixed the specific failures that motivated it (AEGIS delta-verify). If the set is empty (a clean benchmark, or no per-case data), there is nothing for the gate to verify — skip it and proceed.

Step 2: Analyze results

Delegate analysis to the eval-analyzer agent via Task:

Task subagent_type: evaluate-plugin:eval-analyzer
Prompt: Analyze these evaluation results and identify improvement opportunities.
  Skill: <path to SKILL.md>
  Benchmark: <benchmark.json contents>
  Mode: comparison (if baseline data exists) or benchmark (otherwise)

The analyzer produces categorized suggestions:

• instructions: Execution flow improvements
• description: Better intent-matching text
• examples: Missing or insufficient examples
• error_handling: Missing edge cases
• tools: Better tool configurations
• structure: Organizational improvements

Step 3: Filter suggestions

If --description-only, filter to only description category suggestions.

Sort remaining suggestions by priority (high > medium > low).

Step 4: Present suggestions

Present the categorized suggestions to the user:

## Improvement Suggestions: <plugin/skill-name>

Current pass rate: 72%

### High Priority

1. **[instructions]** Add explicit error handling for missing git config
   Evidence: eval-003 fails because the skill doesn't check for git user.name

2. **[description]** Add "conventional commit" as trigger phrase
   Evidence: Skill not selected when user says "make a conventional commit"

### Medium Priority

3. **[examples]** Add breaking change example to execution steps
   Evidence: eval-004 inconsistently handles breaking changes

### Low Priority

4. **[structure]** Move flag reference to Quick Reference table
   Evidence: Flags scattered across multiple sections

If --apply is NOT set, stop here.

Delta-verify gate (required before any apply)

Never write an edit to the live SKILL.md until the drafted candidate has shrunk the source-failure set captured in Step 1 — a higher aggregate pass rate is not sufficient, because a candidate can lift the golden set while leaving every motivating failure broken. Apply only when delta = (source failures before) − (source failures after) is > 0; --force-apply overrides and records the override.

Run the gate against the drafted candidate under eval-results/candidates/, never against the live SKILL.md. Full procedure: references/delta-verify-gate.md.

Step 5: Apply changes (if --apply)

Use AskUserQuestion to let the user select which suggestions to apply:

Which improvements should I apply?
[x] Add error handling for missing git config
[x] Add trigger phrases to description
[ ] Add breaking change example
[ ] Restructure flag reference

If --best-of N with N > 1, follow Step 5a to pick the winning revision first, then continue with the apply flow below using the winner's content.

Draft the approved edits into a candidate file and run them through the Delta-verify gate above. Only proceed to write the live SKILL.md when the gate passes (or --force-apply is set). For each approved suggestion:

1. Read the current SKILL.md
2. Apply the change using Edit
3. Update the modified date in frontmatter

Step 5a: Generate and rank candidates (if --best-of N > 1)

Generate N alternative drafts and let evaluation pick the winner, ranking by source-failure delta first and mean golden-set pass rate second, so a candidate that lifts the aggregate while leaving the motivating failures broken never wins. Treat --best-of without a number as N=3. Candidate generation, the ranking and no-evals fallback, and the Step 5b history entry that records it: references/best-of-ranking.md.

Step 6: Suggest re-evaluation

After applying changes, suggest:

Changes applied. Run `/evaluate:skill <plugin/skill-name>` to measure improvement.

Agentic Optimizations

Quick Reference