# Skill Actually Works

> Measure whether an installed skill changes the output at all, by running the same tasks with it enabled and disabled. Use when auditing an installed skill library, deciding which skills to keep, or when someone claims a skill improved results. Refuses to credit a skill with an effect it cannot demonstrate.

- Skill: `ityaadiii/skill-actually-works` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ityaadiii/skill-actually-works`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ityaadiii/skill-actually-works/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ityaadiii (https://skillmd.com/u/ityaadiii)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ityaadiii/skill-actually-works

---


# Does the skill do anything

People install forty skills and can tell you what none of them changed. A skill that
never fires, or fires and produces what the model would have produced anyway, is
occupying context and attention for nothing.

This is the same question as "does the model matter", asked one layer up, and it gets
answered the same way: by ablation.

## The refusal

**Do not credit a skill with an improvement measured on one example.** If the on/off
comparison has too few tasks to distinguish from noise, the answer is "no measurable
effect on n tasks", not "it works".

## Procedure

1. **Collect 15 to 30 real tasks** the skill claims to cover. Real ones, from actual
   work. Tasks written to suit the skill will flatter it.

2. **Run each task twice**, with the skill available and with it removed. Same model,
   same phrasing, same everything else. Order matters less than keeping it constant.

3. **Check it fired at all.** A large share of skills never trigger, because the
   description does not match how anyone phrases the request. If it did not fire, that
   is the finding, and the fix is the description, not the body.

4. **Grade the pairs blind.** Whoever grades should not know which output had the skill.
   Self-grading a skill you wrote is not evidence.

5. **Use a paired test.** The same tasks went through both conditions, so compare the
   disagreements, not the totals. See `model-bakeoff` for the machinery.

6. **Separate three outcomes.** Changed and better. Changed and worse. Did not change.
   The third is the most common and the least reported.

## Output contract

- tasks run, and how many times the skill actually fired
- pairs where output changed, in either direction
- blind-graded win / loss / tie, with a paired p-value
- verdict: keep, rewrite the description, or remove
- for a "no effect" verdict, say plainly that the skill can be deleted

## What this prevents

A library of forty skills where six do the work, twelve never fire, and the rest are
decoration nobody has the evidence to delete.