Does the skill do anything
People install forty skills and can tell you what none of them changed. A skill that
never fires, or fires and produces what the model would have produced anyway, is
occupying context and attention for nothing.
This is the same question as "does the model matter", asked one layer up, and it gets
answered the same way: by ablation.
The refusal
Do not credit a skill with an improvement measured on one example. If the on/off
comparison has too few tasks to distinguish from noise, the answer is "no measurable
effect on n tasks", not "it works".
Procedure
Collect 15 to 30 real tasks the skill claims to cover. Real ones, from actual
work. Tasks written to suit the skill will flatter it.
Run each task twice, with the skill available and with it removed. Same model,
same phrasing, same everything else. Order matters less than keeping it constant.
Check it fired at all. A large share of skills never trigger, because the
description does not match how anyone phrases the request. If it did not fire, that
is the finding, and the fix is the description, not the body.
Grade the pairs blind. Whoever grades should not know which output had the skill.
Self-grading a skill you wrote is not evidence.
Use a paired test. The same tasks went through both conditions, so compare the
disagreements, not the totals. See model-bakeoff for the machinery.
Separate three outcomes. Changed and better. Changed and worse. Did not change.
The third is the most common and the least reported.
Output contract
- tasks run, and how many times the skill actually fired
- pairs where output changed, in either direction
- blind-graded win / loss / tie, with a paired p-value
- verdict: keep, rewrite the description, or remove
- for a "no effect" verdict, say plainly that the skill can be deleted
What this prevents
A library of forty skills where six do the work, twelve never fire, and the rest are
decoration nobody has the evidence to delete.
1---2name: skill-actually-works3description: Measure whether an installed skill changes the output at all, by running the same tasks with it enabled and disabled. Use when auditing an installed skill library, deciding which skills to keep, or when someone claims a skill improved results. Refuses to credit a skill with an effect it cannot demonstrate.4---56# Does the skill do anything78People install forty skills and can tell you what none of them changed. A skill that9never fires, or fires and produces what the model would have produced anyway, is10occupying context and attention for nothing.1112This is the same question as "does the model matter", asked one layer up, and it gets13answered the same way: by ablation.1415## The refusal1617**Do not credit a skill with an improvement measured on one example.** If the on/off18comparison has too few tasks to distinguish from noise, the answer is "no measurable19effect on n tasks", not "it works".2021## Procedure22231. **Collect 15 to 30 real tasks** the skill claims to cover. Real ones, from actual24 work. Tasks written to suit the skill will flatter it.25262. **Run each task twice**, with the skill available and with it removed. Same model,27 same phrasing, same everything else. Order matters less than keeping it constant.28293. **Check it fired at all.** A large share of skills never trigger, because the30 description does not match how anyone phrases the request. If it did not fire, that31 is the finding, and the fix is the description, not the body.32334. **Grade the pairs blind.** Whoever grades should not know which output had the skill.34 Self-grading a skill you wrote is not evidence.35365. **Use a paired test.** The same tasks went through both conditions, so compare the37 disagreements, not the totals. See `model-bakeoff` for the machinery.38396. **Separate three outcomes.** Changed and better. Changed and worse. Did not change.40 The third is the most common and the least reported.4142## Output contract4344- tasks run, and how many times the skill actually fired45- pairs where output changed, in either direction46- blind-graded win / loss / tie, with a paired p-value47- verdict: keep, rewrite the description, or remove48- for a "no effect" verdict, say plainly that the skill can be deleted4950## What this prevents5152A library of forty skills where six do the work, twelve never fire, and the rest are53decoration nobody has the evidence to delete.
ityaadiii