Bs Skill Auditor

Skill Auditor

Audit and evaluate installed agent skills for quality, freshness, and structural integrity. This is a meta-skill: it treats other skills as its subject matter.

Boundary Rules

These constraints apply to the entire skill. Read them before any execution steps.

• Do not modify any skill file during audit. This skill is read-only.
• Do not execute any skill during audit. This is static analysis, not dynamic testing.
• If a skill directory contains no SKILL.md and only a .gitkeep, report it as V-S4 (orphaned stub) with anchor 0 for all perspectives.
• If a directory under the skills root contains files but no SKILL.md (and is not just a .gitkeep), flag it as V-S5 (unrecognized layout) with anchor 25 and route to manual review.
• If the skills directory is empty or inaccessible, report anchor 0 overall and stop.
• If auto-routing would create a cycle (e.g., bs-skill-auditor -> bs-skill-forge -> bs-skill-auditor), break the cycle by routing to manual review instead.
• Self-audit bias guard: when the audited set includes bs-skill-auditor itself, tag every finding on it SELF-AUDIT, cap its overall anchor at 75, and add to the report preamble: "bs-skill-auditor cannot fully audit itself — an auditor shares its own blind spots. Independent review recommended." Never report a clean 100 for yourself.
• Save the report to docs/reviews/bs-skill-auditor/<YYYY-MM-DD>-health-report.md. If the directory does not exist, create it.

Red Flags — Audit Rationalizations

These are the auditor's own shortcuts, named before they occur. (Pattern: anti-pattern-pre-naming.)

Progressive Disclosure

This skill is organized in three loading tiers:

• Tier 1 -- Metadata only: The orchestrator reads name and description from frontmatter to decide whether to invoke this skill. The full body is not loaded at decision time.
• Tier 2 -- Full body: On invocation, read from Boundary Rules through Execution Steps. This is the complete audit logic.
• Tier 3 -- Reference files: Files under docs/patterns/ are loaded on demand, only when a specific pattern needs deeper inspection during evaluation (e.g., to verify correct pattern application in Perspective 4).

Patterns Applied

This skill itself uses the following patterns from docs/patterns/README.md. When auditing other skills for Pattern Alignment (Perspective 4), these serve as the reference for correct application:

Multi-Perspective Review Panel

Evaluate each skill from four independent perspectives. Assign a discrete confidence anchor to each finding. Do not combine perspectives into a single score -- keep them separate.

Perspective 1: Structural Integrity (S)

Check frontmatter completeness, required sections, file organization.

Why structure matters: Frontmatter parsing failures are among the most common and most silently-degrading issues in agent skill systems. If name or description is malformed, the orchestrator may fail to route to the skill at all -- the skill effectively does not exist. Body size bounds prevent both trivial stubs (< 500 bytes) and context-overwhelming monoliths (> 5000 words).

• YAML frontmatter present with opening --- delimiter
• Frontmatter has closing --- delimiter (second occurrence after line 1). Verify with: awk 'NR>1 && /^---$/{found=1; exit} END{exit !found}' SKILL.md
• Required fields: name (kebab-case), description (starts with "Use when")
• Body length between 500 bytes and 5000 words
• No orphaned .gitkeep files in skill directory
• SKILL.md is the only markdown file in the skill root (references go in subdirectories)

Perspective 2: Safety (Y)

Check for dangerous patterns, secrets, destructive commands.

• No hardcoded API keys, tokens, or passwords in skill body
• No destructive filesystem commands (rm -rf /, chmod 777 /, etc.) without explicit user confirmation gates
• No instructions to bypass permission prompts or security hooks
• No commands that download and execute remote scripts without user review

Perspective 3: Freshness (F)

Check staleness signals.

• Skill has been exercised within the last 90 days. Check with: (1) git log --since='90 days ago' -- <skill-dir>/ 2>/dev/null first; (2) if git returns zero results or fails (e.g., repo has no commits), fall back to find <skill-dir> -name 'SKILL.md' -mtime -90 2>/dev/null; (3) if both fail, mark as "unverified" rather than "stale". A skill is "exercised" if it has been modified, reviewed, or had its test prompts run within the window.

  Why 90 days? Skills rot in two ways: (1) their referenced APIs/tools change, and (2) the agent runtime's behavior evolves. A skill untouched for 90+ days may silently degrade even if the file itself is unchanged. The 3-tier check (git -> find -> unverified) prevents false positives: if we cannot measure freshness, we report "unverified" rather than "stale," because assuming staleness when you cannot verify it erodes trust in the health report.

• Referenced external sources are still accessible. Check with: curl -sI <url> | head -1 (should return HTTP 200). For GitHub repos, use: gh api repos/<owner>/<repo> --jq '.updated_at'. If curl or gh is unavailable, note the check as "unverified" in the report.
• No references to deprecated APIs, removed CLI flags, or retired tools
• Source attributions in docs/patterns/ are still valid (upstream repos exist and are active)

Perspective 4: Pattern Alignment (P)

Check correct usage of documented patterns.

• Tool-first: if auditing this repository (i.e., tools/pattern-alignment.js exists at the repo root), run node tools/pattern-alignment.js — it resolves each declared pattern to its file under docs/patterns/ and reports drift. Use its output as the P findings.
• Fallback (foreign skill directories): each pattern name used in the skill appears in docs/patterns/README.md. Verify with: grep -o '<pattern-name>' SKILL.md for each claimed pattern. Name matching verifies naming only, not correct application — when this is the only check performed, cap the P anchor at 75 and note "name-match only" in the finding.
• The skill cites at least one source for each pattern used (source must appear in the patterns README source column)
• Patterns are used at the correct depth tier (Lightweight/Standard/Deep) for the skill's declared tier

Why pattern alignment matters: Patterns are this project's core methodology (STUDY -> EXTRACT -> DEVELOP). A skill that claims to use patterns but misattributes or misapplies them undermines the entire pattern-library investment. This perspective ties each skill back to the evidence base in docs/research/.

Confidence Anchors

Use discrete anchors only. Never emit continuous scores like "7.3/10".

Assign a confidence anchor to each perspective independently. Then assign an overall anchor: the minimum of the four perspective anchors. A skill is only as healthy as its weakest dimension.

Why min()? Health is a conjunctive property -- all dimensions must be adequate. A skill with perfect structure (S=100) but a hardcoded secret (Y=0) is not healthy. The min() function ensures no dimension can be "averaged away" by strong scores elsewhere. Safety failures tend to produce anchor 0 (V-Y1 blocks processing), so the min() naturally gives safety the highest effective weight without needing explicit weighting math.

Example: S=75, Y=50, F=25, P=50 -> Overall=25 (dragged down by staleness).

Verification Rules + Auto-Routing

When a check fails, route to the appropriate remediation path automatically.

Why these specific rules?

• V-Y1 (hardcoded secrets) is BLOCK because once a secret reaches a skill file, it likely also reached version control, agent context windows, and shipping logs — recovery requires revoking and rotating, not editing. Continuing the audit on a leaking skill spreads the problem.
• V-S3 (>5000 words) is manual review (not auto-fix) because oversized skills usually need conceptual splitting, not mechanical truncation. Auto-truncating would amputate logic.
• V-F1 (>90 days unused) routes to refresh-or-deprecate because skills rot from API drift, tool changes, and pattern evolution at roughly that timescale; older skills silently start producing wrong output.

Self-Review Checklist

Before emitting the final health report, run this checklist on the report itself.

1. Placeholder scan: Search the report for "TODO", "TKTK", "???", "placeholder". Remove or resolve all.
2. Anchor consistency: Every finding has exactly one confidence anchor. No anchors are missing. No finding has multiple anchors.
3. Completeness: Every skill in the scanned directory appears in the report. No skill was silently skipped.
4. Actionability: Every FAIL finding has a corresponding auto-route or manual recommendation. No orphaned failures.
5. Tone: The report describes issues factually, without judgmental language. It does not praise or blame -- it states what is and what to do.
6. Self-application: Run this skill against its own SKILL.md, applying the self-audit bias guard from Boundary Rules (tag findings SELF-AUDIT, cap overall at 75). Note any self-flagged issues in the report preamble. Fix root-cause issues before the next audit cycle.

Execution Steps

Step 1: Scan

The skills root defaults to the skills/ directory relative to the repository root. If scanning from a different location, accept it as an explicit parameter.

List all directories under the skills root. A skill directory is one that contains (or should contain) a SKILL.md file. Exclude directories named .git, node_modules, docs, evaluation, external, tools.

For each skill directory, record:

• Skill name (from directory name)
• Whether SKILL.md exists
• Last modification date (via git log --since='90 days ago' -- <skill-dir>/ 2>/dev/null, falling back to find <skill-dir> -name 'SKILL.md' -mtime -90 2>/dev/null; mark "unverified" if both fail)

Step 2: Evaluate

For each skill, run the four-perspective review. Use the checklists above. Record every finding with its perspective tag (S/Y/F/P), pass/fail status, confidence anchor, and the verification rule ID if applicable.

Assign per-perspective confidence anchors during evaluation. These feed directly into the overall score computation in Step 3.

Step 3: Score

Compute the overall anchor as min(S, Y, F, P) using the anchors already assigned during evaluation in Step 2.

Step 4: Route

For each FAIL finding, apply the auto-routing table. Generate the remediation path. Check for circular routing before emitting recommendations (see Boundary Rules).

Step 5: Report

Produce the health report in this format:

SKILL HEALTH REPORT
===================
Scan date: [date]
Skills scanned: [count]
Skills passing (overall >= 50): [count]
Skills failing (overall < 50): [count]
Skills unassessable (anchor 0): [count]

PER-SKILL DETAILS
-----------------
[skill-name]
  Structural: [anchor] [pass/fail findings, comma-separated rule IDs]
  Safety:     [anchor] [pass/fail findings, comma-separated rule IDs]
  Freshness:  [anchor] [pass/fail findings, comma-separated rule IDs]
  Pattern:    [anchor] [pass/fail findings, comma-separated rule IDs]
  Overall:    [anchor]
  Remediation: [list of auto-routes and manual recommendations]

SUMMARY
-------
Top remediation priorities (by severity):
1. [V-Y findings first]
2. [V-S findings second]
3. [V-F findings third]
4. [V-P findings fourth]

If more than 20 skills are scanned, include full per-skill details only for skills with overall anchor < 75. List all other skills in a compact summary line: [skill-name]: Overall [anchor], all perspectives >= [min-perspective].

Step 6: Self-Review

Run the self-review checklist on the report. Fix any issues. Then emit the final report.

If all skills pass with overall anchor >= 50 and no V-Y findings, conclude the report with: "No remediation required. All skills are healthy."

Depth Tier

This skill is Standard tier. It is a meta-skill used periodically (not daily), with moderate failure cost (a bad health report wastes review time but does not break production).

Test prompts

Three prompts that exercise this skill end-to-end, mirrored in evaluation/datasets/batch-1-test-prompts.json.

1. Happy path -- audit a real skill: "Audit the bs-social-card skill for health. Run the full four-perspective review and produce a health report." -- expected: scans skills/bs-social-card/SKILL.md, produces a report with S/Y/F/P anchors, identifies bs-social-card as a valid skill with frontmatter, assigns overall anchor, lists any findings with rule IDs, saves report to docs/reviews/bs-skill-auditor/<date>-health-report.md.

2. Edge -- scan all skills in a batch: "Run bs-skill-auditor on the entire skills/ directory. There should be ~8 skills -- give me the compact summary for any that score >= 75." -- expected: scans all skill directories, excludes .gitkeep-only stubs (reports them as V-S4 with anchor 0), applies compact mode threshold (>= 75 gets summary line, < 75 gets full detail), safety override ordering visible in the SUMMARY section.

3. Adversarial -- audit a deliberately broken text fixture: "Audit this provided Skill text without writing it to disk: it has no frontmatter closing delimiter, contains API_KEY=sk-abc123def456ghi789jkl012mno345pqr678stu901 inline in the body, and has a body under 100 bytes. Do NOT skip the safety check. Do not create, modify, or delete any Skill file while auditing this fixture." -- expected: V-Y1 triggers BLOCK before any other perspective is processed; the hardcoded secret is detected from the provided text; the fixture is reported with Y=0 and overall=0; the other perspectives may be unprocessed because the safety override blocks them; the report clearly flags the security finding in the preamble; no Skill file or directory is created, modified, or deleted at any point.