Does the skill do anything
People install forty skills and can tell you what none of them changed. A skill that never fires, or fires and produces what the model would have produced anyway, is occupying context and attention for nothing.
This is the same question as "does the model matter", asked one layer up, and it gets answered the same way: by ablation.
The refusal
Do not credit a skill with an improvement measured on one example. If the on/off comparison has too few tasks to distinguish from noise, the answer is "no measurable effect on n tasks", not "it works".
Procedure
Collect 15 to 30 real tasks the skill claims to cover. Real ones, from actual work. Tasks written to suit the skill will flatter it.
Run each task twice, with the skill available and with it removed. Same model, same phrasing, same everything else. Order matters less than keeping it constant.
Check it fired at all. A large share of skills never trigger, because the description does not match how anyone phrases the request. If it did not fire, that is the finding, and the fix is the description, not the body.
Grade the pairs blind. Whoever grades should not know which output had the skill. Self-grading a skill you wrote is not evidence.
Use a paired test. The same tasks went through both conditions, so compare the disagreements, not the totals. See
model-bakeofffor the machinery.Separate three outcomes. Changed and better. Changed and worse. Did not change. The third is the most common and the least reported.
Output contract
- tasks run, and how many times the skill actually fired
- pairs where output changed, in either direction
- blind-graded win / loss / tie, with a paired p-value
- verdict: keep, rewrite the description, or remove
- for a "no effect" verdict, say plainly that the skill can be deleted
What this prevents
A library of forty skills where six do the work, twelve never fire, and the rest are decoration nobody has the evidence to delete.