Skill Evaluation Iteration
Core Workflow
- Define the target behavior and realistic user scenarios before editing the skill.
- Run or simulate a baseline without the skill when practical.
- Run or simulate the same scenario with the skill.
- Compare outputs by decision quality, missing steps, correctness, concision, safety, tool use, validation, and deliverable usefulness.
- Patch the skill to close observed gaps. Keep changes small and domain-specific.
- Validate the skill folder with the official validator.
- Repeat only when the new test reveals a material gap.
Output Contract
For skillset improvement work, return:
- Scenarios evaluated
- Rubric used
- Gaps found
- Skills changed
- Validation run
- Remaining risks or deferred improvements
Patch Decision Rule
Patch a skill only when the test reveals a concrete trigger, workflow, output, validation, freshness, or routing gap. Do not edit skills just to make them longer.
Evaluation Criteria
- Trigger accuracy: skill activates for the right requests and avoids unrelated ones.
- Metadata sync:
agents/openai.yaml still reflects the current SKILL.md purpose, especially after major edits.
- Account routing: account-level instructions mention the skill when it should be globally discoverable.
- Context efficiency: SKILL.md is concise and references are loaded only when needed.
- Procedural value: skill changes the agent's workflow, not just wording.
- Quality bar: output includes acceptance criteria, validation, and domain-specific checks.
- Robustness: skill handles edge cases, constraints, and missing context.
- Maintainability: skill avoids stale facts unless it includes a freshness rule.
- Hygiene: generated scaffold placeholders, TODOs, stale examples, and irrelevant boilerplate are removed.
When To Forward-Test
Forward-test when the skill is complex, high-impact, or repeatedly used. Use realistic prompts and raw artifacts. Avoid leaking expected answers into the test prompt.
References
- Read
references/skill-evaluation-checklist.md when evaluating or improving a skill.
- Read
references/skill-evaluation-scenarios.md when selecting reusable scenarios for a broader skillset evaluation pass.
1---2name: skill-evaluation-iteration3description: Evaluate, forward-test, and improve Codex skills. Use when Codex is asked to test a skill, compare performance with and without a skill, improve a skillset, make skills stronger, validate skill usefulness, create acceptance tests for skills, or iterate skill instructions based on realistic scenarios.4license: MIT5---6
7# Skill Evaluation Iteration
8
9## Core Workflow
10
111. Define the target behavior and realistic user scenarios before editing the skill.
122. Run or simulate a baseline without the skill when practical.
133. Run or simulate the same scenario with the skill.
144. Compare outputs by decision quality, missing steps, correctness, concision, safety, tool use, validation, and deliverable usefulness.
155. Patch the skill to close observed gaps. Keep changes small and domain-specific.
166. Validate the skill folder with the official validator.
177. Repeat only when the new test reveals a material gap.
18
19## Output Contract
20
21For skillset improvement work, return:
22
23- Scenarios evaluated
24- Rubric used
25- Gaps found
26- Skills changed
27- Validation run
28- Remaining risks or deferred improvements
29
30## Patch Decision Rule
31
32Patch a skill only when the test reveals a concrete trigger, workflow, output, validation, freshness, or routing gap. Do not edit skills just to make them longer.
33
34## Evaluation Criteria
35
36- Trigger accuracy: skill activates for the right requests and avoids unrelated ones.
37- Metadata sync: `agents/openai.yaml` still reflects the current `SKILL.md` purpose, especially after major edits.
38- Account routing: account-level instructions mention the skill when it should be globally discoverable.
39- Context efficiency: SKILL.md is concise and references are loaded only when needed.
40- Procedural value: skill changes the agent's workflow, not just wording.
41- Quality bar: output includes acceptance criteria, validation, and domain-specific checks.
42- Robustness: skill handles edge cases, constraints, and missing context.
43- Maintainability: skill avoids stale facts unless it includes a freshness rule.
44- Hygiene: generated scaffold placeholders, TODOs, stale examples, and irrelevant boilerplate are removed.
45
46## When To Forward-Test
47
48Forward-test when the skill is complex, high-impact, or repeatedly used. Use realistic prompts and raw artifacts. Avoid leaking expected answers into the test prompt.
49
50## References
51
52- Read `references/skill-evaluation-checklist.md` when evaluating or improving a skill.
53- Read `references/skill-evaluation-scenarios.md` when selecting reusable scenarios for a broader skillset evaluation pass.
gabrielmoreira