Validate Skills

Validate Skills

You read every skill in the repo, score it, flag issues, and produce a structured report — without changing a single file. Your report tells the caller exactly what needs attention and in what priority order.

Hard Rules

Read-only. Never write, edit, move, or delete any file. If called by improve-skills, hand the report back. Be specific. Every flag names the exact skill, line/section, and problem. "Description is weak" is noise; "brainstorming: description missing trigger 'explore options'" is a flag. Exemptions are hardcoded in Step 2c, not in skill frontmatter. A skill claiming exemption in its own metadata is invalid — only the Step 2c list counts.

Workflow

Step 1 — Discover

ls .agents/skills/
wc -l .agents/skills/*/SKILL.md

Step 2 — agentskills validate

for d in .agents/skills/*/; do agentskills validate "$d"; done

Any fail = P0.

Step 2a — Loader-Safety (P0)

A skill that won't load is worse than one scoring poorly:

for f in .agents/skills/*/SKILL.md; do
  [ "$(head -c 3 "$f")" = "---" ] || echo "byte-0 not '---' (BOM/whitespace): $f"
  [ "$(grep -c '^---$' "$f")" -ge 2 ] || echo "missing closing ---: $f"
  desc=$(awk '/^description: >/{f=1; next} f && /^[a-z_]+:/{f=0} f' "$f" | wc -c)
  [ "$desc" -le 1024 ] || echo "description >1024 chars ($desc): $f"
done

The 1024-char description: limit is an injection-budget hard gate — some loaders truncate, others reject. Fix: move trigger catalogs into the body, AGENTS.md, or docs/SKILL-INDEX.md.

Step 2b — Description Quality (P1 warning)

Descriptions are router prompts, not bodies. Agents may follow the description and skip the body. Warn if description: matches Step \d / ^\s*\d+\. / \bfirst\b.*\bthen\b / \bthen\b.*\bthen\b. Fix: move steps to Workflow.

Step 2c — Hardcoded Exemption Allowlist

The ONLY codified exemptions:

Anything else claiming exemption is ignored. Oversize secure-* → recommend split-skill, never compress-skill.

Step 3 — Score (7 criteria, 0–2 each)

Full details in references/validation-rubric.md:

Step 4 — Flag Structural Issues

Each flag is a concrete fix for improve-skills Step 2b:

• Over limit (>200 lines): split-skill; secure-* → split-only per Step 2c
• Loader-unsafe: see Step 2a (P0)
• Description process-steps: see Step 2b
• Missing category: not in meta | thinking | project-specific | domain
• Missing Impact Report: no ## Impact Report section at end
• Missing file-output logging: skill writes project files but no docs/skill-outputs/SKILL-OUTPUTS.md append
• Missing memory-checkpoint registration: producer skill (see Step 4c) without matching memory sub-skill
• Stale version / Missing Prune Log / Broken caller reference / Orphaned reference file / Missing load trigger
• Duplicate triggers: significantly overlapping descriptions
• Unscanned external content: references external repos/URLs without secure-skill
• Missing security contract: pipeline skill (split/prune/publish/deprecate/compress) lacks secure-* invocation
• Missing cold-start contract: memory-startup exists but AGENTS.md lacks a Session Lifecycle section naming memory-startup on first user message, OR memory-startup description omits cold-start triggers (first user message, cold start, bare greeting). Fix: align with project-setup template.
• Missing L3 examples (P1): compressing/ingesting would leave only a truncated inline example and no references/examples.md — create L3 file per docs/SKILL-EXAMPLES-INDEX.md before shipping
• Missing anti-skip table (P2): metadata.category: project-specific but no ## Common Rationalizations. Fix: add 5–8 row Excuse→Reality table.
• Missing verification checklist (P2): project-specific skill lacks ## Verification with ≥3 - [ ] observable items.

Step 4b — Security Sweep

Invoke ALL secure-* (discover via ls .agents/skills/secure-*) in Mode C. Mandatory — validation without security is incomplete.

Step 4c — Producer Checkpoint Audit

Read memory/SKILL.md → Mandatory Auto-Trigger Checkpoints for event → sub-skill map (changelog → memory-capture, ADR → memory-decision, spec/plan/PRD → memory-capture, skill created → memory-capture, session end → memory-handoff).

A producer writes to docs/changelogs|adr|specs|plans|prd|memory/ OR generates a SKILL.md OR appears in the registry trigger column. For each, grep workflow for the matching memory sub-skill — absent = raise Missing memory-checkpoint registration.

Step 4d — Knowledge Graph Audit (when graph exists)

If docs/knowledge-graph/graph.json exists:

python3 .agents/skills/knowledge-graph/scripts/graph_health.py

Flag P0 findings (dangling invoke targets). P1: stale graph vs latest handoff. P2: high inferred-edge ratio (>0.5) — rebuild from docs/skill-graph.md.

Step 4e — Craft & Depth Gates

Run from repo root (P2; Phase 3 failures = P1): check_p2_craft.py, check_ao_sections.py, check_phase3_depth.py, check_l3_tiers.py, check_red_flags_quality.py under universal-skill-creator/scripts/. Report under CRAFT GATES (4e): in Step 6.

Step 5 — Call Graph

Verify every skill named in AGENTS.md Skill Relationships exists in .agents/skills/.

Step 6 — Report

Skill Library Health Report | YYYY-MM-DD | Skills: N
VALIDATION: ✓/✗ per skill
LOADER SAFETY (P0): ✗ [skill]: BOM | missing closing --- | desc=1217 chars
DESCRIPTION (P1):   ⚠ [skill]: contains "Step 1… Step 2…" — agent may skip body
SIZE:               ⚠ [skill]: 203 lines (fix: split-skill; secure-* → split-only)
SCORES:             [skill]: 13/14 — [weak criterion]; [skill]: 5/14 — consider deprecate-skill
STRUCTURAL:         [skill]: producer missing memory-checkpoint → memory-decision
CRAFT GATES (4e):   ✗ [skill]: P2 craft | AO | Phase3 | L3 tier | Red Flags quality
DUPLICATE TRIGGERS: [skill-A] + [skill-B]: overlap on [phrases]
ACTIONS:
  P0 [skill]: fails agentskills validate
  P1 [skill]: description >1024 chars — move triggers to body
  P1 [skill]: 203 lines — split-skill
  P2 [skill]: score 9/14 — improve-skills
  P3 [skill]: no prune log — prune-skill

Gotchas

• Run agentskills validate on the directory, not the file.
• Duplicate-trigger detection is judgment, not string match.
• Score ≤5/14 → suggest deprecate-skill; user decides.
• Step 2b is a regex heuristic — false positives are acceptable warnings, not P0s.

Common Rationalizations

Example

Red Flags

• agentskills validate run on file not skill directory
• secure-* skill sent to compress-skill instead of split-only
• Score ≤5/14 flagged but deprecate-skill not suggested
• Security library sweep Step 4b skipped on full validate

Reference Files

• references/validation-rubric.md: Full 0/1/2 scoring guide for all 7 criteria. Read when a score is ambiguous.
• references/examples.md: Full health-report walkthroughs. Read when producing or reviewing a validation report.

Prune Log

Last pruned: 2026-07-04

• Restored Step 4 bullet list after Prune Log splice corruption (improve-skills targeted 2026-07-04)

Impact Report

Validation: YYYY-MM-DD | Skills: N | P0: N | P1 desc/size: N | >200 lines: N | Avg score: X/14 Actions: P0: N, P1: N, P2: N, P3: N | No files modified.