Skill Stocktake

skill-stocktake — Skill Quality Audit

Evaluate installed skills and assign each a verdict: Keep / Improve / Update / Retire / Merge, plus Out of scope for skills this harness does not own (Phase 0). This skill does NOT do the improving — once it has a verdict, it hands off to skill-creator (the improvement engine). That boundary is the point: stocktake is the quality gate, skill-creator is the fixer.

Design note (v3.1). Per-item quality/freshness and set-level existence/overlap are different properties. Deterministic checks are code-owned and unconditional; per-item judgment gets narrow, fresh contexts; set-level judgment gets a light description sweep with targeted deep reads. Historical verdict counts and named past outcomes stay in git history, not in this operational prompt: prior outcomes anchor a new audit toward reproducing them instead of judging the current library.

Modes ($ARGUMENTS)

changed detects changes inline (no script):

find ~/.claude/skills -name SKILL.md -newermt "$(jq -r .evaluated_at ~/.claude/skills/skill-stocktake/results.json)"

Phase 0 — Structural pre-pass (deterministic, code-owned)

Run BEFORE any LLM judgment. These checks never ride on per-item attention:

1. skill-health scan — dangling references / missing artifacts across the library (run its scanner per ~/.claude/skills/skill-health/SKILL.md). Mandatory, not optional: this is the structural layer of Curate (ADR-0019 layering), and the v2.2 verification showed exactly these defects slipping through judgment-owned checks.
2. Ledger hygiene — for every results.json entry: the path must exist on disk and the key must follow the canonical rule below. Dedup violations (same path under two keys → keep the canonical key, delete the duplicate). Entries whose path no longer exists are removed (note them in the report as retired-from-disk).
3. Existence before judgment — a skill that fails the existence check never reaches Phase 2. Do not let an LLM assign Keep to a file that is not there (this happened: two nonexistent paths carried Keep verdicts in the 2026-07-05 ledger).
4. Ownership before judgment — the same skill-health run reports external: skills whose directory is a symlink out of the skills root (a package manager's tree, a sibling checkout). These do not get an Improve/Update verdict. Record them as Out of scope with the owner and real path in the reason, and route any real defect upstream (issue / PR). A verdict here is not actionable — an edit lands in someone else's tree, never reaches git, and disappears on their next upgrade. Live miss on 2026-07-25: hunk-review drew an Improve for a stale flag table and the fix was written into /opt/homebrew/Cellar/hunk/0.17.1/…; it was reverted and filed as modem-dev/hunk#595. The check is fully structural (is_symlink()), so it belongs in code, never in per-item attention.

Phase 1 — Inventory

Enumerate skill definition files with Glob (no script needed):

• ~/.claude/skills/*/SKILL.md
• if cwd has .claude/skills/, also {cwd}/.claude/skills/*/SKILL.md (project skills)

Canonical ledger keys: one key per skill directory name, matching the path on disk (ADR-0047). A rule or skill referencing a name in prose does NOT make it a skill.

Usage evidence (parent-owned; batch agents never see it). Step 0 — run the script and transcribe the numbers; do not re-derive them by hand:

uv run --project ~/.claude/skills/skill-stocktake python -m scripts.usage_stats --days 14

JSON out, exit 0 always: counts (per skill — deliberate / slash / invoke / last_used, ordered by deliberate use), window_label, measurable, and excluded (how many rows each correction removed). The four corrections that decide whether the number means anything — the event-type split, the two sandbox exclusions, and the real-span label — live in the script's docstring together with their measured effect on the real log. Do not re-implement them as a jq one-liner here (ADR-0052): the verify-bootstrap case is why — 2 apparent uses, 0 after correction.

Two readings the script hands you but cannot make for you:

• measurable: false → render usage as — (unmeasured). Never render it as 0 — unmeasured and unused are different facts.
• span_shorter_than_window: true → label the column with window_label (the log's real span), not the nominal 14d.

Reference-URL evidence (parent-owned for the same reason, plus one of its own: Phase 2 runs batch agents in parallel, and fetch named URLs inside each of them is a burst against the same hosts — the shape rules/common/debugging.md forbids). Check every URL the library names once, serially, here:

set -o pipefail   # without it a producer crash reads as "0 dead links, all healthy"
uv run --project ~/.claude/skills/skill-health python -m scripts.scan_refs --external-urls |
  uv run --project ~/.claude/skills/skill-health python -m scripts.url_liveness --urls-from -

Extraction is scan_refs's job, not a grep: it skips fenced code blocks and template slots, so the audit does not fetch its own example URLs. (--offline on url_liveness is the dry run — it reports every URL skip without touching the network.)

Verdicts are live / dead / blocked / skip. blocked is not dead — a 403 is bot policy, not absence; reporting it as dead sends the next reader chasing a link that is fine. skip, offline_suspected: true, halted: true, and a non-null source_error all mean unchecked, not fine — and input_lines: 0 means nothing was ever checked, which is not the same as a clean corpus. If the run halted on a rate limit, say so in the report and do not re-run the remainder in this session.

State the scan result up front: which paths were scanned, how many skills found, and whether usage is measurable.

Phase 2 — Per-item scrutiny (parallel small batches, fresh contexts)

Split the target set into batches of 10–12 files and launch one subagent per batch in parallel. Small batches are the point: per-item attention dilutes as a context fills. Do NOT pass prior verdicts or the ledger to batch agents (anchoring); do NOT pass usage data (parent-owned dimension).

Each batch agent applies, per skill:

Stage 1 — binary screen. Explicit Yes/No per item; surface only the No answers (these six questions are the canonical set — skill-creator §4 reuses them for its creation-time draft gate by reference, not by copy):

• Actionability: concrete steps/commands/examples you can act on?
• Scope fit: name, trigger (description), and body aligned — not too broad or narrow?
• Within-batch uniqueness: no content overlap with other skills in this batch? (a documented orchestrator/sub-skill split is NOT overlap; cross-batch overlap is Phase 3's job, not this agent's — the same goes for contradictions with skills outside this batch)
• Currency: unconditionally verify every artifact the skill names — ls each referenced path (~/.claude/agents/<name>.md, hooks, bundled scripts) and run --help / version checks for named CLI flags. "Verify if it looks stale" is banned phrasing: the condition is what dilutes. Deterministically checkable claims get deterministic checks, every time. Do not fetch URLs — the parent checked them once in Phase 1 and hands you the verdicts; parallel batch agents each fetching is a burst.
• Hygiene: is the body free of bloat — trivial prohibition lists, repeated emphasis, and step-by-step recitals that could fold into a principle (skill-creator §3 の書き方)? Do NOT fold greppable detection terms, self-enforcing prohibitions, or numeric thresholds — abstraction destroys their function (ADR-0058 rejected uniform shortening for exactly this). Bodies grow after creation; this question is the standing check the creation-time gate cannot repeat.
• Description audit: is the description free of standing instruction text — NOT-for routing, "Use PROACTIVELY"-style fire directives, mentions of other skills — beyond stating what the skill is for and when to reach for it? A listed description resides in the system prompt every session as an unaudited instruction layer (RFC-0018). Quote each instruction sentence found (semantic judgment, not regex); do NOT judge its effectiveness — the parent joins quotes with invoke stats at synthesis.

Stocktake-only existence pass (every item, including Keep-bound skills). This is separate from the canonical six-question quality screen above, which skill-creator reuses. Answer both questions explicitly:

• Standalone value: if this file disappeared, would the library lose a user job that is not already covered by an installed skill, rule, runtime substrate, or cheap on-demand generation?
• Cost asymmetry: does the skill's independent trigger and reusable judgment justify its selection, drift, and maintenance cost?

A No answer makes the skill a non-Keep candidate and must be pressure-tested. Do not shield a skill from this pass merely because it is current, well written, recently used, or part of a documented layer.

Stage 2 — verdict pressure-test (non-Keep candidates only). Generate 1–3 skill-specific atomic yes/no questions that try to refute the draft verdict before finalizing it. Answer each with one line of evidence (file read, path check, WebSearch). A refuted defect → fall back toward Keep. A confirmed defect → the No answers become the concrete improvement list handed to skill-creator (Improve/Update) or the removal rationale (Retire).

Evaluation is holistic judgment, not a numeric rubric — binary answers are evidence feeding the verdict, never aggregated into a score (a satisfaction ratio changes no decision and dilutes a single dominant No). Batch agents return structured verdicts with self-contained reasons. Every skill gets the fixed existence pass; only non-Keep candidates get the dynamic pressure-test questions.

Phase 3 — Overlap and contradiction probe (one dedicated agent, set-level)

Duplication and contradiction are set properties; they get a specialist, not a side effect. A per-item batch cannot see either — each agent reads only its own 10 files and finds nothing wrong with any of them individually. One agent, whole library:

1. Read name + description (+ heading structure) of every target file — light pass.
2. Propose candidate clusters greedily (false positives are fine at this stage).
3. For each candidate cluster, read the bodies side by side and judge:
   • ABSORBABLE_OVERLAP — two skills serve the same user intent, trigger, and job; maintaining both adds selection/drift cost; the source has named unique residue that can move into a named Merge target
   • FULLY_ABSORBED — another asset already covers the whole user job and the source has no unique residue to move; name the covering asset and use Retire, not Merge
   • CONTRADICTION — two skills that can both load give opposite instructions for the same situation, or a sub-skill's content violates the canon it defers to. Quote both sides. See the contradiction checks below.
   • DOCUMENTED_LAYERING — orchestrator→sub-skill, rule→skill, declared defer; quote the defer line and name each file's independently triggerable responsibility
   • ADJACENT_BUT_DISTINCT — near domain, different job; name one concrete independent user request that each member handles
4. One extra pass over ~/.claude/rules/ and MEMORY.md: is a rule re-stating a skill (or vice versa) beyond a declared pointer? Flag promotion residue (a skill whose content a rule has fully absorbed) as Retire/Merge candidates.

Contradiction checks (a declared boundary is a claim, not evidence)

DOCUMENTED_LAYERING certifies "not duplicated". It does not certify "not contradictory", and a quoted defer line is the start of the check, not the end. For every cluster labelled DOCUMENTED_LAYERING, run all three:

• Does the subordinate file comply with the canon it defers to? Read the canonical file's rules and look for the subordinate stating a different rule for the same decision — especially defaults ("if the user gives no X, use Y"), which contradict quietly because they only fire in the unspecified case.
• Is the defer bidirectional? A one-directional defer (canon claims the subordinate, subordinate says nothing) means the subordinate loads alone without its correction whenever its own description wins the trigger. A broad description on a subordinate file turns a latent contradiction into an active one.
• Do the two files presuppose the same author / project model? Imported skills (origin = an external repo) carry their author's premises. A conflict of premises reads as a normal-looking instruction and survives every per-file check.

ABSORBABLE_OVERLAP produces Merge verdicts because named residue must move into the target. FULLY_ABSORBED produces Retire because the target already covers the whole job and there is nothing to move. CONTRADICTION produces Improve (add the missing defer, delete the conflicting rule, narrow the trigger) or, when the premises themselves clash and the subordinate never fires on its own, Retire-and-absorb — name what must move before deletion.

Phase 4 — Synthesis (parent)

Merge Phase 2 verdicts, Phase 3 overlap / contradiction verdicts, and parent-owned usage data:

• Content owns the verdict; usage is reference evidence only. Render the 14-day deliberate-use count and last-used date, but never make usage a prerequisite, veto, threshold, or automatic candidate trigger. Recent use does not immunize invalid content; zero use does not condemn valid content.
• Retire content-first. A skill can be Retire regardless of usage when (a) its workflow or underlying premise is obsolete, (b) another asset actually absorbs its user job and no standalone residue remains, or (c) it is too thin, non-actionable, or cheaply regenerated for its selection/drift/maintenance cost. State which route the evidence supports and what covers the need instead.
• Treat cadence only as a clue. A zero in 14 days may prompt a trigger/cadence explanation, but it never creates a candidate or verdict. A unique seasonal skill may remain Keep. Conversely, a recently used skill whose content is fully replaced may be Retire.
• Separate existence from freshness. A stale version pin, retired CLI flag, or dead pointer that is locally fixable is Update, not Retire. It becomes Retire only when the verified evidence shows the workflow/premise itself is no longer worth preserving.
• Charge aggregate cost through the existence pass. Holding a skill adds selection, drift, and maintenance cost. Do not protect a merely adequate skill because it has no individual defect; ask whether its independent trigger and reusable judgment still justify a separate file. This is judgment, never a quota.
• Description-audit quotes join usage here. An instruction sentence whose aim (self-firing, misfire routing) the deliberate-use record does not support is a removal candidate: fold it into the body or an existing rule. A supported trigger surface is left alone (do not strip descriptions that demonstrably fire).
• Conflicts (e.g. batch says Keep, probe says Merge) are resolved by the parent reading the cited evidence, not by vote.

Evaluation is origin-blind: the same checklist applies to every skill.

Render a table: Skill | 14d | last used | Verdict | Reason, where the count is deliberate use only (slash + invoke). If the log is younger than 14 days, replace 14d with its real span.

Phase 5 — Consolidation

Confirm one by one (config-gc's confirm-each design): walk the non-Keep candidates sequentially — for each, show the evidence first, then ask [y/n/skip]. Never batch the approval. The user can stop at any point; skip records the verdict in the ledger unactioned.

• Retire / Merge: per file, present (1) the specific defect found, (2) what covers the same need instead, (3) the impact of removal (dependent skills, rule pointers, MEMORY references) → ask [y/n/skip]. Act only after the user confirms that file.
• Improve / Update: offer per skill — "Hand <skill> to skill-creator? [y/n/skip]" — and on approval invoke skill-creator with the target skill and the No-answer list. Stocktake never does the improvement work itself.
• Update the ledger: Read results.json → merge this run's verdicts → Write it back (evaluated_at = real UTC from date -u +%Y-%m-%dT%H:%M:%SZ). In changed mode, preserve the prior verdicts of skills you did not re-evaluate.
• If MEMORY.md exceeds 100 lines, propose compression.

Reason quality (required)

Every reason must be self-contained — decision-enabling on its own. "unchanged" alone is banned; always restate the evidence. For non-Keep verdicts, the reason cites the No answers from the binary screen / pressure-test (question + one-line evidence).

• Retire: state the defect + the replacement. Bad: "Superseded" / Good: "disable-model-invocation: true already set; continuous-learning-v2 covers the same patterns plus confidence scoring. No unique content remains."
• Merge: name the target + what to integrate. Bad: "Overlaps with X" / Good: "42-line thin content; Step 4 of chatlog-to-article already covers this workflow. Integrate the 'article angle' tip there as a note."
• Improve: which section, what change. Bad: "Too long" / Good: "276 lines; 'Framework Comparison' (L80–140) duplicates ai-era-architecture-principles. Delete it to reach ~150 lines."
• Keep (mtime-only change in changed mode): restate the original rationale.

results.json (lean ledger)

{
  "evaluated_at": "2026-06-24T11:09:00Z",
  "skills": {
    "<skill-name>": {
      "path": "~/.claude/skills/<name>/SKILL.md",
      "verdict": "Keep",
      "reason": "...",
      "mtime": "2026-01-15T08:30:00Z"
    }
  }
}

A ledger for verdict history and the last-audit timestamp only. Update it inline with Read/Write, not a script. Global skills only (project skills are read fresh from cwd, not cached here). Phase 0's hygiene rules (canonical keys, existing paths) are part of every write.

Related

• skill-creator — the improvement engine; hand off Improve/Update work to it.
• skill-health — the deterministic structural layer; Phase 0 runs its scanner as a mandatory pre-pass (enumerate-then-decide: code enumerates debt, judgment decides).
• config-gc — GC over skill existence and the whole of ~/.claude; stocktake judges skill quality.
• rules-stocktake — the same audit for ~/.claude/rules/ (residency cost instead of usage).
• agent-stocktake — the third sibling, for ~/.claude/agents/ (hybrid cost model: description = residency, body = invocation).
• generation-audit — on a model-generation change, collects runtime-layer evidence (conflict / redundancy / drift) and hands the skills slice to Phase 4 synthesis as external evidence (read, never require).
• harness-boundary — design-time lens (layer / portability / obsolescence) for proposed mechanisms; applied to an installed skill, its Delete / Move are Phase 4 evidence only.
• repo-asset-stocktake — the same stocktake pattern for a project repo's non-code assets.
• llm-as-judge — the generic judge design canon (binary screen → pressure-test → holistic named verdict, no aggregation); Phase 2 is its library-scale implementation.
• harness-sync — use it to sync this skill to its public repo.
• Usage measurement: ~/.claude/hooks/log-skill-usage.sh → ~/.claude/metrics/skill-usage.jsonl (a measurement layer independent of stocktake).

References

The two-stage binary-question design (screen → verdict pressure-test, holistic verdict, no score aggregation) follows the checklist-decomposition evaluation line: BinEval "Ask, Don't Judge" (arXiv:2606.27226), CheckEval (arXiv:2403.18771), TICK (arXiv:2410.03608). Scores are deliberately not adopted: BinEval's own limitations show over-decomposition degrades correlation on holistic quality dimensions, and a satisfaction ratio would dilute a single dominant No.

The v3.1 hybrid architecture separates deterministic verification, narrow per-item scrutiny, and set-level existence/overlap judgment. The split preserves attention for each property without carrying prior verdict outcomes into the next audit.