Graph Audit

Graph Audit

Concept of the skill

Graph audit has three evidence planes: authored source (SKILL.md plus audit-state.json), compiled projection (manifest samples and exports), and executable or report gates (lint, drift, protocol consistency, schema constants, routing eval, application-eval shape, audit-manifest, conformance ledger, inline-count guard, Markdown links, and preflight).

Coverage

• Schema conformance — canonical SKILL.md metadata validates against schemas/SKILL_METADATA_PROTOCOL_schema.json: required v8 identity fields, subject/public enums, name shape, and name-directory parity.
• Sidecar consistency — audit-state.json validates against schemas/skill-audit-state.schema.json, and audit/eval/provenance fields live in the sidecar instead of being duplicated as hand-authored SKILL.md state.
• Two-file contract (ADR-0019) — a skill is SKILL.md (the 25 agent-facing frontmatter fields) plus its sibling audit-state.json (the audit/eval/provenance fields), governed by two distinct schemas. scripts/check-schema-constants.js asserts BOTH required sets (the 5 core frontmatter fields and the 7 required sidecar fields) and reads schema_version + the four verdicts from the sidecar — a field placed in the wrong file is a real finding even when each file independently parses.
• Manifest sync / joined projection — examples/skills.manifest.sample.json and generated manifests are the joined model of SKILL.md + audit-state.json, not a stale copy of either file alone (node scripts/generate-manifest.js --validate-only).
• Protocol/schema parity — field-reference docs, manifest-mapping docs, schema constants, sidecar fields, and sample manifests agree about the live contract (check-protocol-consistency.js, check-schema-constants.js).
• Relation integrity — every target under relations.* corresponds to a real sibling skill directory and uses the right predicate semantics (related/broader/narrower/boundary/disjoint_with/verify_with/depends_on). No deterministic gate currently enforces target existence. scripts/skill-lint.js deliberately does NOT walk relation targets (that check was removed in the 2026-05-19 lint reduction — it now only emits advisory boundary warnings, checks 5b/5c), and scripts/generate-manifest.js copies the relations block through unmodified (generate-manifest.js:360-382); its --validate-only pass validates the compiled manifest against schemas/manifest.schema.json, which checks each relation item's shape, not whether its target resolves to a real skill. So a dangling edge is surfaced only by the audit pipeline's graded Relation quality surface (lib/audit/skill-audit.js:550) and the human-facing fix-hint at lib/audit/skill-audit.js:413 that recognizes a "does not match any known skill" message — both semantic/graded, not a CI gate. An auditor must therefore resolve relation targets by hand (glob the sibling library) rather than trusting manifest:validate to do it. This live ownership gap also exposes stale docs (see § Known Stale Claims).
• Suppresses-edge orientation — scripts/skill-lint.js emits advisory warnings (checks 5b/5c) when a relations.suppresses reason reads as deference ("use X instead") rather than ownership ("I own this over X"), or when a suppresses edge crosses subject shelves (a cross-domain distinction belongs in anti_examples + relations.related, not an exclusion guard).
• Drift baselines / truth-source freshness — each skill's recorded drift_check.truth_source_hashes still match the live truth-source files via scripts/skill-graph-drift.js; this is the mechanism behind truth_verdict. A truth source that changed since last_verified without a re-recorded hash is silent staleness, and a live drift result is stronger evidence than a stale sidecar truth_verdict: PASS.
• Routing-eval honesty — routing_eval: present is backed by a current routing-harness run (scripts/skill-graph-routing-eval.js) over a manifest generated from current source, with positive examples routing to the skill and anti-examples blocked or explained.
• Eval-artifact coherence — eval_artifacts: present and graded verdict claims are backed by real gradeable artifacts. scripts/check-audit-manifest.js enforces that a graded comprehension_verdict (PROVISIONAL/PASS/SHALLOW/REDUNDANT) requires evals/comprehension.json on disk, and a high-stakes graded application_verdict (APPLICABLE/MIXED/HARMFUL) requires evals/application.json.
• Application-eval shape — scripts/check-application-evals.js validates each evals/application.json against schemas/application.schema.json: the mode: "application" discriminator, the ≥5-case floor, the per-case required-field set, unique ids, and the red-herring recommendation.
• Understanding-fields gate — when a skill declares comprehension_state: present, it must carry the five flat top-level Understanding fields (mental_model, purpose, boundary, analogy, misconception); the legacy nested concept block is accepted for back-compat only, and the content must contain no out-of-scope repo-specific nouns. This is audit checklist item 8; scripts/skill-audit-preflight.js reports whether the fields an op needs (including these) are present before audit/improve/evaluate runs.
• Corpus-level conformance — scripts/skill-conformance-ledger.js turns the binary lint gate into a per-shelf "X / N clean" number plus an error-type histogram — the incremental signal the all-or-nothing gate cannot give while the corpus is draining through the audit loop.
• Inline corpus-count hygiene — corpus counts ("N canonical skills") must be generated in docs/status.generated.md + the live find command, never hand-stamped into prose; scripts/check-inline-skill-counts.js (npm run counts:check) fails the build on any inline count.
• Markdown-link integrity — scripts/check-markdown-links.js catches dangling intra-repo links (paths and anchors) emitted by skill bodies and docs — a broken [...](path) is the same silent-failure class as a dangling relation.
• Grounding & project fit — every non-empty project[] skill has a populated grounding declaration with subject_matter, grounding_mode, truth_sources, failure_modes, and evidence_priority. The schema's allOf requires only grounding for a project target; project[] is an optional belonging-entity affiliation, not a requirement.
• Name-directory parity — a skill's name field matches the name of its parent directory (required for SKILL.md compatibility).
• Public/export safety — project-specific audit skills (this one included) are treated as graph-internal unless the export gate intentionally includes them; check for stale/private/provenance fields before public distribution.

Philosophy of the skill

Skill graphs fail silently. A broken relation, stale truth-source hash, or drifted enum value does not crash the agent — it changes what the router loads, what the manifest exports, or what the auditor trusts. That kind of failure is worse than a crash because the system still looks healthy. The audit's job is to turn every silent bug into a loud one before the graph accumulates enough drift that agents can no longer trust its edges.

A graph audit is a constellation of single-purpose checkers, not one mega-linter. This is a deliberate design choice: scripts/skill-lint.js was reduced (2026-05-19) to enforce only external-mandate, canonical-source constraints — valid YAML, schema validation, identifier shape, non-empty description, name/directory parity, and subjects[0]/subject agreement (plus advisory boundary-orientation warnings). Everything else — manifest parity, eval coherence, drift, routing — lives in dedicated tools (the drift sentinel, the manifest validator, the routing engine, the audit-manifest checker, the application-eval checker, the conformance ledger). Auditing well means knowing which checker owns which invariant and running the right one, not assuming lint covers everything. The sharpest example of this trap is relation-target existence: it is owned by no deterministic checker today (see Coverage → Relation integrity), so an auditor who assumes a green lint/manifest:validate proves every edge resolves will ship a dangling edge.

Two integrity surfaces sit side by side. The binary gate (npm run lint / npm run verify) goes red if any skill fails, which is the correct ship-blocker but gives no incremental signal mid-migration. The conformance ledger (skill-conformance-ledger.js) is report-only and turns the same per-skill linter into a measurable percentage so progress is visible while the corpus drains. Use the gate to block a release; use the ledger to watch the backlog move.

The audit's reason to exist is upstream-confirmed: the base Agent Skills spec is still intentionally minimal — only name and description are required, with license, compatibility, metadata, and allowed-tools optional, and metadata treated as arbitrary additional data that downstream runtimes (e.g. OpenCode) ignore when unknown. No upstream release has absorbed the Skill Metadata Protocol's typed scope, relations, grounding, or eval state, so every invariant this skill audits is a Protocol addition layered on the minimal base — which is exactly why a dedicated audit discipline is needed: the upstream spec will never check these for you.

Mental Model

Treat the library like a small database system across three evidence planes:

• Authored source. SKILL.md is the authored capability row (routing contract, scope, relations, grounding, agent-facing guidance); audit-state.json is the sidecar state row (owner, freshness, drift hashes, eval state, routing coverage, the four verdicts).
• Compiled projection. The manifest is a materialized view: it joins skill source with sidecar state and derives summary facets. The marketplace export is a second projection (the Agent-Skills-compatible shape).
• Executable / report gates. Relations are foreign keys plus routing semantics (related = adjacency, boundary = score-aware exclusion when this skill wins, depends_on/broader expand loading, verify_with = cross-check). The audit commands are integrity checks against those planes — but some questions are intentionally runbook-owned rather than standalone-gate-owned.

Do not close a finding on the wrong plane. A linter pass does not prove truth-source freshness. A sample-manifest routing pass does not prove the canonical source still has the same relations. A sidecar PASS verdict does not outweigh a fresh drift command showing changed truth sources.

Key Files

Verification Matrix

skill-lint.js is the first smoke test, not the whole audit. A skill that lints clean can still have a dangling relation, a drifted truth source, a stale manifest, or a dishonest verdict — those are caught by the other checkers, and a dangling relation target is caught by NO deterministic checker, so resolve those by hand. Use the smallest command that owns the question.

# Structural SKILL.md + sidecar checks for one skill
node scripts/skill-lint.js graph-audit --no-color
node scripts/skill-lint.js --path ../skills/skills graph-audit   # explicit root for the sibling library
node scripts/skill-lint.js --include-template                    # also lint the example template

# Truth-source freshness for one skill directory
node scripts/skill-graph-drift.js ../skills/skills/quality-assurance/graph-audit

# Manifest projection and schema validation (NOT relation-target existence)
node scripts/generate-manifest.js --validate-only

# Protocol constants and schema/docs/mapping parity
node scripts/check-schema-constants.js
node scripts/check-protocol-consistency.js

# Routing-eval honesty for this skill against a fresh manifest fixture
node scripts/skill-graph-routing-eval.js --skill graph-audit --manifest examples/skills.manifest.sample.json --confusion-matrix

# Audit/eval evidence consistency + application-eval shape
node scripts/check-audit-manifest.js
node scripts/check-application-evals.js --skill graph-audit --check

# Corpus conformance triage + volatile-count hygiene + Markdown links
node scripts/skill-conformance-ledger.js --json
node scripts/check-inline-skill-counts.js
node scripts/check-markdown-links.js

# Audit-loop readiness before comprehension/application work
node scripts/skill-audit-preflight.js graph-audit --for all

Interpretation rules:

• Exit code 0 on skill-lint.js means structural checks passed. It does NOT prove drift freshness, manifest sync, routing behavior, or audit-artifact readiness.
• Exit code 1 on skill-graph-drift.js means at least one local truth source changed or broke relative to the sidecar hash baseline. Treat this as current truth evidence even if the sidecar still says truth_verdict: PASS.
• generate-manifest.js --validate-only can fail because of unrelated corpus skills. Separate per-skill graph-audit findings from corpus-wide migration debt — and remember it validates manifest shape, never relation-target existence.
• A routing-eval pass is only meaningful when the manifest being tested was generated from the current canonical SKILL.md and current sidecar. A stale sample can hide relation or health-state drift.
• check-audit-manifest.js proves graded verdict claims have artifacts. It does not certify usefulness; application_verdict: APPLICABLE is the behavior-certification signal.
• check-application-evals.js --check proves application-eval JSON is structurally conformant. It does not prove the grader has run or that the application verdict is useful.
• skill-conformance-ledger.js is the incremental corpus-drain signal: per-shelf clean-count + error-type histogram help prioritize cleanup; the plain report is not a binary gate unless a threshold flag is used.
• check-inline-skill-counts.js proves active docs are not hand-stamping volatile corpus totals — generated status/conformance docs own the live numbers.
• check-markdown-links.js proves local Markdown targets and anchors resolve. It does not decide whether a skill's H2/H3 body structure is pedagogically complete; use the audit-loop runbook for that.

Choose the umbrella gate by work mode:

npm run verify          # full gate — corpus lint + manifest validation + routing eval + export (CONTENT-inclusive)
npm run verify:system   # SYSTEM-only gate — schema constants, protocol, docs, fixtures, marketplace, unit tests
npm run release:check   # the release-critical corpus gates verify:system OMITS (manifest:validate, routing-eval, export, marketplace:verify)

verify:system green does not mean publishable: it deliberately omits the corpus gates, so it can stay green while manifest:validate / routing-eval are red (e.g. skills missing a required scope). Before claiming the library is publishable, run npm run release:check.

Relation Integrity Procedure

Because no deterministic gate owns relation-target existence, audit relations by hand:

1. Read relations from the canonical SKILL.md, then compare against the generated manifest entry to catch stale projection.
2. Confirm every relation predicate is semantically correct: related is browse/routing adjacency; boundary excludes listed skills from co-routing when this skill wins; verify_with names cross-check context; depends_on/broader expand loading; disjoint_with is mutual exclusion.
3. Check relations.suppresses entries for ownership wording — a good reason says "graph-audit owns metadata-integrity verification over X"; a weak reason says "use X instead."
4. Check target existence against the sibling skill library or generated manifest data yourself. Do NOT claim skill-lint.js proves relation targets exist (the live lint only owns relation shape via schema validation plus boundary reason/cross-domain warnings), and do NOT claim generate-manifest.js --validate-only proves target existence (it copies relation fields through and schema-validates shape). If no deterministic command owns the target-existence check, report that as a tooling gap rather than claiming a gate already proved it.
5. Run routing eval when anti-examples or boundary behavior are part of the claim.

Known Stale Claims (current audit findings)

Two repo docs currently assert relation-target validation that the live code does not perform — both are real findings a graph audit should raise (schema/code wins over prose per the Doc Ownership Map precedence):

• docs/QUICKSTART-30MIN.md (§ "Break the relation and watch lint catch it") states the lint "walks every relation predicate … and verifies that every named target resolves to a real sibling skill." False against current scripts/skill-lint.js (relation-target checking was removed in the 2026-05-19 lint reduction; only advisory boundary checks 5b/5c remain).
• skill-metadata-protocol/SKILL_METADATA_PROTOCOL.md § Relations states "the manifest compiler refuses to emit a relation to a skill that does not exist" and to "rely on npm run manifest:validate for the relation-integrity guarantee." False against current scripts/generate-manifest.js, which copies relations through unmodified (:360-382) and validates only manifest shape (:845-857).

Record both as documentation drift until a deterministic relation-target checker is added or the docs are corrected. The general lesson generalizes: a sidecar truth_verdict: PASS recorded on an earlier date is an audit record, not live truth — re-run skill-graph-drift.js and trust the live result when they disagree.

Evals

This skill has a local comprehension artifact at evals/comprehension.json and a Skill Graph fixture at examples/evals/graph-audit.json. The local artifact exercises the portable concept (sidecar ownership, drift versus lint, manifest joins); the Skill Graph fixture keeps repo-grounded prompts tied to current protocol files, sidecar state, manifest joins, routing examples, relation metadata, and gradeable-artifact evidence.

No evals/application.json exists yet. The sidecar's eval_artifacts: present (ADR-0019 places eval_artifacts in audit-state.json, not frontmatter) is disk-truth meaning an eval file exists on disk — here the comprehension artifact — NOT a claim that an application eval exists; both comprehension_verdict and application_verdict stay honestly UNVERIFIED. The moment a graded application_verdict is claimed, check-audit-manifest.js will require an evals/application.json on disk — so do not stamp a graded application verdict without authoring (and running) that artifact in the same change. Before claiming comprehension/application readiness, run preflight: comprehension readiness requires the sidecar state plus the five flat Understanding fields and a sufficiently sized comprehension eval; application readiness requires a gradeable application artifact that passes the application-eval shape gate.

Verification Checklist

• SKILL.md passes schema validation and name-directory parity.
• audit-state.json validates and owns audit/eval/provenance state rather than duplicating those fields in SKILL.md.
• Generated manifest entries match the joined SKILL.md + audit-state.json source model (generate-manifest.js --validate-only).
• Protocol docs, field mappings, schemas, and schema constants agree (check-protocol-consistency.js, check-schema-constants.js).
• Drift command agrees with the truth-source hashes recorded in the sidecar, or the sidecar is updated by the audit loop (skill-graph-drift.js).
• Relation predicates use the correct semantics, and relation targets have been checked by hand against real sibling skills (no deterministic gate; the graded Relation-quality surface + manual glob are the only coverage — do not overstate lint/generator ownership).
• Routing eval uses a fresh manifest and proves both positive examples and anti-example boundaries (skill-graph-routing-eval.js).
• Graded comprehension/application verdicts have matching gradeable artifacts and receipts (check-audit-manifest.js).
• evals/application.json meets the schema floor (check-application-evals.js) when application readiness is claimed.
• Every skill with comprehension_state: present carries the five flat Understanding fields with no out-of-scope repo-specific nouns.
• non-empty project[] skills have a complete grounding declaration (and project affiliation where the protocol requires one).
• No dangling intra-repo Markdown links or anchors (check-markdown-links.js); body H2/H3 structure findings are tied to the audit-loop runbook, not overstated as schema failures.
• No corpus count is hand-stamped into prose (check-inline-skill-counts.js); skill-conformance-ledger.js consulted when the question is corpus-wide cleanup priority.
• docs/QUICKSTART-30MIN.md and SKILL_METADATA_PROTOCOL.md § Relations do not continue to claim lint/manifest walks all relation targets — or that stale claim is recorded as doc drift.
• Public/export fixtures do not leak project-internal or private data and do not expose repo-specific skills unintentionally.
• Every skill's name field matches its parent directory name (SKILL.md compatibility).

Verification

After applying this skill, verify:

• The task matches the declared scope, coverage, or positive examples.
• The response follows this skill's workflow or checks instead of generic advice.
• The exclusions in ## Do NOT Use When do not point to a better skill.

Do NOT Use When