Skill Infrastructure

Skill Infrastructure

Coverage

• The library-as-database mental model: skill-infrastructure as the linter, integrity checker, and query planner for a SKILL.md library
• Why deterministic, zero-LLM tooling is mandatory for skill libraries (trustworthy enough for CI gates; LLM-based health checks are circular)
• The five categories of skill-health tooling: (1) inventory and frontmatter validation, (2) protocol consistency, (3) conflict detection (overlap, imperative, code duplication, heading overlap), (4) routing health (gap analysis, miss tracking), (5) drift detection (truth-source hashing, mirror parity)
• Eval quality patterns: minimum eval threshold per skill, the contradiction-check eval type, the negative-expectation requirement, valid eval-type taxonomy
• Imperative conflict detection: the same-target-opposite-polarity rule, three-check false-positive suppression, when conflicts indicate scope-tightening vs merging
• Routing gap analysis: how to read a "routing-misses" log, how to distinguish keyword gaps from skill content gaps, signal-hygiene rules to suppress noise
• Maintenance workflows: when to run a full health check, the order in which to run the categories, what to fix before what
• Anti-patterns that cause invisible decay: dirty-tree manifest writes, deletion-as-conflict-resolution, eval-renumbering during cleanup, scope misuse to mask threshold violations
• The verification gate before any batch skill commit: every category clean, every new skill meets eval minimums, every routing change is reflected in the manifest
• Package and workspace-root integrity: the npm CLI entrypoint must dispatch to the same scripts as local development while resolving schemas from the package and skills/manifests from the caller workspace

Philosophy

A skill library is only as useful as its worst skill. When agents load stale, conflicting, poorly-routed, or mirror-drifted skills, they get worse at tasks — not better. A skill library at scale (50+, certainly 200+) decays invisibly: eval counts drift below minimums, keyword maps miss whole product areas, mirror copies fall out of sync, and two skills start giving opposite instructions for the same function.

The infrastructure exists to catch that decay deterministically, with zero LLM calls. Every health check reads files, parses structure, and computes — it does not reason, infer, or hallucinate. That makes the output trustworthy enough to use in CI-style gates. An LLM-based health check is circular: the same probabilistic component that produces the skill library cannot reliably grade its own output.

If the skill library is a codebase, skill-infrastructure is its combined linter, type-checker, and dead-code detector — always-on, zero-model, CI-safe.

The maintenance commitment is: run a full health check after any batch skill work, fix threshold violations before they reach the gate, and document conflicts before the agents loading those skills are affected.

The Library-as-Database Mental Model

Treat the skill library as a database and skill-infrastructure as its query planner, integrity checker, and migration-safety net combined.

Every script in this domain reads files and computes — it does not reason. Output is deterministic, reproducible, and safe to embed in CI gates.

The Five Categories of Skill-Health Tooling

A production skill library needs all five. Missing any one allows a class of decay through.

1. Inventory and Frontmatter Validation

Walks the skill tree, parses every SKILL.md's frontmatter, and validates against the schema.

Reference implementation: scripts/skill-lint.js in this repo runs T5 per-skill lint covering most of these.

2. Protocol Consistency

Cross-checks that the schema, generator, sample manifest, and field-reference docs all agree.

Reference implementation: scripts/check-protocol-consistency.js runs C1–C7 protocol checks.

3. Conflict Detection

Pairwise comparison across all active skills to surface duplication, contradiction, and structural overlap.

Imperative extraction patterns:

• Positive: \b(?:always|must|required|mandatory)\b\s+(.{10,80})/gi, \buse\s+(.{5,60})\s+(?:for|when|instead)/gi
• Negative: \b(?:never|do\s+not|don't|must\s+not|prohibited)\b\s+(.{10,80})/gi, \bdo\s+NOT\s+use\s+(.{5,60})/gi

Three-check false-positive suppression for the most common phantom-conflict sources (e.g. Do NOT use X matching as positive use X):

1. Lookbehind check — if not, never, don't, or cannot appears in the 20 characters before a positive match, suppress it
2. Within-match check — if not appears between the start of the match and the target identifier (extracted via backticks), suppress it
3. Same-line dedup — if the target was already extracted as a negative on the current line, suppress any positive extraction for that target

When a conflict is real vs spurious:

4. Routing Health

Reads the routing-miss log and the routing config to identify where the keyword map fails.

Output sections:

1. Keyword gaps — words appearing in routing-misses but absent from the active keyword map, sorted by frequency
2. Eval failure breakdown — skills with 2+ failures, grouped by failure category
3. Suggested actions — add_keyword for frequent gaps; improve_skill for skills failing for content reasons

Signal hygiene rules (suppress these from the suggestion list):

• Low-signal tokens (v, v4, skill, start, events, daily, log, status, health)
• Single-word misses already covered by an existing multi-word phrase (e.g. error should not surface if error recovery is already mapped)
• Stale historical misses where the full query already routes under the current config
• Synthetic no-match probes used to verify the router's miss path
• Bundle entries (only direct keywordMap entries participate in free-text matching)

Reference implementation: the routing harness (scripts/skill-graph-routing-eval.js in this repo) provides the negative case — verifying that a skill's authored examples actually route to it. A real routing-gap report is the symmetric positive case (verifying that real user prompts reach a skill).

5. Drift Detection

Detects when the world the skill claims has changed without the skill being updated.

Reference implementation: scripts/skill-graph-drift.js in this repo records and verifies truth-source hashes against the live files.

Eval Quality Patterns

The minimum-threshold rule

Every active skill should have at least 7 evals. Most healthy skills carry 9–15 covering happy paths, edge cases, anti-patterns, and contradiction checks. The threshold is a floor, not a goal.

Recommended enforcement:

• Error if eval_count < 3 (or warn for scope: reference skills, where the file is documentation more than testable behaviour)
• Warn if eval_count < 7

Below 7, the skill is statistically under-tested. Below 3, it is effectively un-evaluated.

The contradiction-check eval type

A contradiction-check eval tests that the agent correctly handles a documented exception or boundary condition that a simpler reading of the skill would mishandle. Format:

{
  "id": 5,
  "type": "contradiction-check",
  "grounding": "repo-specific",
  "difficulty": "adversarial",
  "prompt": "Skill A says always use the scoped fetcher. One service uses the unscoped fetcher with an inline comment. Is this wrong?",
  "expected_output": "Not wrong — the unscoped fetcher with an inline justification comment is the documented exception for system-level reads.",
  "expectations": [
    "Correctly identifies the documented exception from the skill's anti-patterns table",
    "Does NOT flag the usage as a bug without reading the inline comment",
    "Distinguishes a system-level exception from a regular violation in application code"
  ]
}

Use a contradiction-check when:

• A skill has a documented exception that overrides the general rule
• Two adjacent skills appear to contradict each other but actually operate in different scopes
• A historical false positive or conflict was resolved and the resolution is non-obvious

The negative-expectation requirement

Every eval case with an expectations array must include at least one expectation containing does not, never, must not, should not, or do not. Without this, evals become pure happy-path tests and miss the failure modes that motivated the skill.

Recommended enforcement: the inventory tool flags any eval missing this pattern in a missingNegativeEvalIds field of its report.

Valid eval types

Maintenance Workflows

The full health check sequence

Run after any batch skill work — creating ≥ 3 skills, changing routing config, or modifying skill content across multiple files. Order matters:

# 1. Inventory snapshot — discover invalid frontmatter, missing fields, broken refs
<inventory tool>

# 2. Protocol consistency — cross-schema, manifest, generated-doc parity
<protocol-consistency tool>

# 3. Conflict detection — overlap, imperatives, code duplication
<conflict tool> --conflicts

# 4. Routing gaps (last 7 days)
<routing-gap tool> --since 7d

# 5. Drift sentinel — truth-source hashes, mirror parity
<drift tool>

# 6. Regenerate manifest + index from the now-clean tree
<manifest generator> --write

Review the output of steps 1–5 before running step 6. The manifest and index should reflect a clean state — never write a manifest from a dirty tree.

Adding evals to an existing skill

1. Read the skill's existing eval set
2. Identify gaps: are all eval types represented? Are there contradiction-checks for documented exceptions?
3. Draft new cases with id, prompt, expected_output, expectations, type, grounding, difficulty
4. Ensure every new case has at least one does not / never / must not expectation
5. Append to the eval array — never renumber existing IDs (eval-history logs reference numeric IDs)
6. Re-run the inventory tool and verify the skill no longer appears in belowMinimum or missingNegativeEvalIds

Triaging imperative conflicts

1. Run the conflict tool with --conflicts --json to get the structured list
2. For each conflict, identify which skill's instruction is more precise or more scoped
3. Resolve by narrowing scope, not by deleting the instruction:
   • "Use X" → "Use X in app code" (narrower)
   • "Always use X" → "Always use X for tenant-scoped data" (specific)
   • Move the negative instruction to an anti-patterns table row that does not start with "Use"
   • Add an inline qualifier (e.g. // system: <reason>) that the detector can distinguish
4. Re-run --conflicts to confirm the conflict is gone
5. Document the resolution in a conflict log so future audits know it was deliberate

Fixing invalid frontmatter

After editing, re-run the inventory tool and confirm the skill no longer appears in the invalid-frontmatter list.

Anti-Patterns

Verification

Before any batch skill commit, verify:

• Inventory tool exits with zero critical errors
• Zero invalid-frontmatter entries
• Zero broken mappings in routing config
• All new skills meet the eval minimum (≥ 7, or ≥ 3 with explicit warning acceptance)
• All new evals include at least one negative expectation
• Conflict tool shows no NEW conflicts vs baseline
• Routing-gap report shows no new keyword gaps caused by your changes
• Manifest generator completed without errors
• Mirror parity verified if the library replicates to multiple harness directories
• Skill index header count matches actual skill count

Do NOT Use When