Optimizing Skills

Optimizing Skills

Treat the skill document as the parameter under optimization: change it only when the change demonstrably beats the version you already ship. This is the discipline distilled from SkillOpt (microsoft/SkillOpt, arXiv:2605.23904) — its training apparatus dropped, its reproducibility discipline kept. The point is to stop editing skills on intuition and start editing them on evidence.

Core principle

A skill edit is only worth shipping if it strictly improves measured behavior on a held-out check. Most edits that feel like improvements don't move the needle, and some quietly regress. The gate below is what separates a real improvement from a confident guess.

The gate — run it every revision

1. Assemble a held-out check set. 3–8 representative tasks/prompts the skill should handle well, and it must include the failure(s) that prompted this revision. Keep the set fixed across the revision so before/after scores are comparable.
2. Hold two versions. best = what you currently ship (never let it silently degrade). candidate = best + your proposed edits.
3. Score both on the check set. "Run" here = dispatch each check task to the Agent tool (subagent_type=general-purpose) with the skill version in context, or evaluate by hand for small sets. Give the scoring agent the skill version and the task, and nothing else. Never hand it the ledger, your revision notes, or the diagnosis that motivated the edit: it will solve the task from those instead of from the skill, and the score stops measuring the skill. WikiSkill (arXiv:2608.27454) ablated exactly this and lost 2.8 points of final quality, 7.8 on their hardest split, by letting the worker read the improver's knowledge store. Score per criterion, not one collapsed pass/fail. When a task carries several criteria, the criterion that decides accept/reject is the failure that prompted this revision; the others are regression guards that must not get worse. Collapsing criteria masks the win: in the down-skilling-v1.2.0 retro, the edit drove architectural hallucination 60%→0% while an unrelated length criterion stayed 0/5 in both arms — a single combined pass/fail scored that as a 0–0 tie and would have rejected a large, real improvement.
4. Accept only if candidate strictly beats best on the triggering-failure criterion, with no regression guard worse. Ties → reject, keep best. An edit that doesn't move the needle does not ship.

When the skill's own output is compiled by an Agent (down-skilling and creating-skill produce a prompt an author writes from the SKILL), score ≥2 author samples per version, or fix one author across both arms. A single author sample per arm lets author capability dominate the edit effect: the same down-skilling edit measured 95%→0% with one author pair and 60%→0% with another — real either way, but n=1 cannot tell a real edit from a lucky author.

This two-tier best/candidate split is the heart of it: a working revision can explore, but the shipped skill only ever ratchets upward.

Bounded edits (the "textual learning rate")

Cap edits per revision — default ~4 distinct add/replace/delete operations, fewer as the skill matures. Large speculative rewrites drift and destroy your ability to attribute a regression to a cause. If a revision wants more edits than the budget, rank and keep the top ones (below) and let the rest wait.

Reflect: failures first, then successes

Separate the evidence before proposing edits:

• Failure reflection. Across the failing cases, find the common, systematic pattern — not a one-off edge case. Propose edits that fix the pattern. Failures take priority in any merge.
• Success reflection. Across cases that already work, find generalizable patterns worth encoding so they survive future edits. Reinforce; don't duplicate.

For both: edits must generalize (never hardcode task-specific values), and must not duplicate content already in the skill — patch genuine gaps only.

Rank when over budget

When candidate edits exceed the budget, keep them in this priority order:

1. Systematic impact — fixes a recurring failure across many cases, not one.
2. Complementarity — fills a real gap rather than restating existing content.
3. Generality — phrased as a durable principle, not tied to one task/entity.
4. Actionability — concrete, followable guidance over vague advice.

Drop the rest. They can return next revision if still warranted.

Protect the hard-won core

If a skill has a battle-tested core that routine edits keep eroding, fence it off and treat it as off-limits to fast edits. Revisit it only on a deliberate longitudinal review: compare the same check tasks across several versions to catch slow drift and regressions that single-edit review misses. (SkillOpt fences this region with HTML-comment markers and only rewrites it at epoch boundaries — the same idea, manual cadence.)

Recording rejected proposals

FIRST action of any revision, before you score anything:

python3 scripts/skill_ledger.py check --skill <name> --before best.md --after candidate.md

Exit 1 means this exact edit was proposed before and failed the gate. Read the recorded criterion and score, then propose something else. Do not re-score it.

After the gate decides, record the outcome — accepted or rejected:

python3 scripts/skill_ledger.py record --skill <name> \
    --before best.md --after candidate.md --outcome rejected \
    --criterion "<the triggering failure>" --score-before 0.6 --score-after 0.6

The ledger computes the diff itself and appends it. A rejected candidate is the artifact worth keeping. The edit reverts and the record stays. The next session cannot see this one, which is why the record has to live outside your context rather than in it.

This is WikiSkill's skill-impact.md (arXiv:2608.27454), whose outer loop keeps a never-rolled-back record of every proposal and its fate while the skill itself rolls back. Their proposer is told in prose not to repeat rejected approaches. check makes that mechanical instead, so it no longer depends on anyone having read the ledger. The script lives in claude-workspace at scripts/skill_ledger.py.

Carry memory across revisions

The ledger above holds the artifacts: which diffs were tried and how they scored. remember() holds the judgment the diffs cannot carry. After a revision, record what you learned about editing this skill — which kinds of edits helped, which were brittle, redundant, or harmful — via remember() tagged with the skill name. Before the next revision, recall() it. This is the compounding part: each revision starts smarter than the last, the way SkillOpt's optimizer-side meta-skill conditions its future edits.

Edit mechanics

Edits are literal string operations (the Edit tool): the target text must match exactly or the edit is a silent no-op. Keep targets unique and verbatim. Prefer append / insert-after-heading / replace-exact / delete-exact, and verify each edit landed before scoring.

When NOT to use this

• Authoring a brand-new skill from scratch → creating-skill.
• Tracking/rolling back versions during development → versioning-skills.
• One-off prose with no reuse → just write it.

Checklist

• Held-out check set assembled (includes the triggering failure), fixed for the revision
• Edits bounded (~4 max), each generalizable and non-duplicative
• Failure patterns addressed before success reinforcement
• Candidate scored per-criterion against best; accept decided by the triggering-failure criterion, others as regression guards; shipped only if strictly better
• For Agent-compiled artifacts (down-skilling, creating-skill): ≥2 author samples per version, or a fixed author across arms
• Hard-won core left untouched unless doing a deliberate longitudinal review
• Candidate run through skill_ledger.py check before scoring; a repeat was dropped, not re-scored
• Scoring agent given the skill version and the task only, never the ledger or the diagnosis
• Gate outcome recorded via skill_ledger.py record, rejections included
• Lesson about editing this skill recorded via remember()

For the deeper "dispatch reflection/scoring to the Agent tool" recipe and the adapted reflection/ranking prompt templates, see references/skillopt-provenance.md.