Drift Audit

Drift audit

Audit the source markdown of registered upstream docs pages against the local Claude config — skills, rules, CLAUDE.md, MCP templates — and emit a structured findings report. Inline execution; the report stays in the conversation so the user can iterate on follow-up actions.

Read-only — never modify any artifact this turn. Rewriting a skill / rule / CLAUDE.md / MCP template based on the findings is a separate, per-artifact task initiated after this audit completes.

1. Sources

The audit input is a registry, not a fixed pair of pages. Read references/sources.md at the start of every run and drive the pipeline from it. Never hardcode a repo, path, section heading, or doc host into the run — if something needed for a source isn't in its registry entry, the entry is incomplete and that is the finding.

Each entry carries an id (what --sources matches), its GitHub repo / branch / path (plus files, when that path is a directory) or plain url, a shape (table / prose / changelog) that drives extraction, sections for the WebFetch fallback, a drill block (host, mechanism, anchor-strip patterns) for Phase 3, and the artifacts classes it can produce findings against. The registry's Shape contracts section defines what "an entry" means per shape; its Adding a source checklist is the procedure for widening the audit.

As registered today: fabric and powerbi (table), vscode-agent and fabric-iq-ontology (prose), and claude-code and skills-for-fabric (changelog). Don't restate that set as a count anywhere — the registry owns it, and this line already went stale once by omitting fabric-iq-ontology for five days after it was registered on 2026-09-02. These are public and fetchable anonymously with two exceptions. claude-code (590 KB) and skills-for-fabric (58 KB) are files with no sections list, so in practice only the github-mcp path can read them. And powerbi's upstream repo was taken down — 404 at the API, web and raw endpoints as of 2026-09-07 — so that source fetches by the strategy written into its own registry entry rather than from a repo field.

2. Argument parsing

The user invokes with a prior reference — slash form /drift-audit <ref>, natural language ("audit since SHA abc1234", "audit since 2026-03-15"), or no argument. The reference is applied to every audited source independently.

• SHA (40-hex or 7+hex prefix) — resolve to its commit timestamp against any registered source's repo (the SHA's source repo doesn't have to match; we only need its date). Use that date as the floor for every source's since window.
• ISO date (YYYY-MM-DD) — use directly as the floor.
• No argument — default to 35 days back from today. Print the resolved floor date in the audit window section.
• Anything else — ask the user to clarify; do not guess.

A single reference applied to every source is intentional: the audit window is "what changed since I last checked," not "what changed in the Fabric repo specifically." Each source resolves the window against its own commits.

--sources <id,id> narrows the run to the named registry ids. Absent, audit every registered source. An id with no registry entry is an error — list the valid ids and stop rather than silently auditing a subset. Skipped sources are named in the report so an empty bucket is never mistaken for a clean one.

3. Read-only scope

Prohibited tools this turn: Edit, Write, MultiEdit, NotebookEdit. Do not call them regardless of what the audit findings suggest is needed.

If the user asks within this same turn to "fix the drift you found", "update fabric-mlv", "rewrite the skill", "patch CLAUDE.md", etc., refuse and explain:

The audit is read-only by design. Each artifact rewrite is its own task with its own context — the audit report you just received is the input to that follow-up. Start a new request naming the specific skill or rule to update.

This separation matters because a single bad rewrite during audit triage can silently corrupt the artifact set, and the per-skill rewrite task has different review and validation steps than the audit itself. The body prompt is the only enforcement — allowed-tools lists the auto-approval scope, not a hard restriction.

4. Phase 1 — Fetch and diff each source

Work through the selected sources in registry order. Two fetch paths; pick once, at the top of the run, and say which one the report used.

An entry may override both. Where a registry entry documents its own fetch strategy — because its upstream repo is gone, or it was never GitHub-hosted — that strategy replaces 4a and 4b for that source, and any trust check it specifies is mandatory rather than advisory. § 1's no-hardcoding rule is what makes the entry authoritative: read it before assuming repo is a static field, and name the path taken in the report as you would for 4a or 4b.

4a. Preferred path — github-mcp

Available when the github-mcp tools are present in the session. Returns exact file bytes and real unified patches, so nothing is reconstructed from a summary.

1. Resolve HEAD — list_commits on the source's repo, filtered to its path, perPage: 1, fields: ["sha"]. Then get_commit on that one SHA with detail: "none" for its date. Record <head-sha> and <head-date>. detail governs file stats only — the commit message always comes back, so on a squash-merged repo budget a few KB for that one call.
2. List commits in window — list_commits with the same path filter, since: <floor-date>T00:00:00Z, and the same fields: ["sha"] (add html_url if the report wants links). Always list with minimal fields. A squash-merged repo puts an entire release branch in one commit, and the message alone then runs to thousands of characters — one filtered page of microsoft/vscode-docs came back at ~74 KB before this was pinned down. Fetch messages only for the SHAs that turn out to matter. If zero commits, mark the source "no changes" and move on (it still gets an audit-window line and a Next-run entry).
3. Get the changes — two strategies, by commit count:
   • 5 or fewer in-window commits — get_commit per SHA with detail: "full_patch" and read the unified patch for the source's path. Added and removed lines come straight from the patch; no side-by-side reconstruction.
   • More than 5 — get_file_contents at the prior ref (resolve it with Resolving the diff base in § 4b — that rule governs this path too, and resolving it by commit order instead of by path silently drops changes) and again at <head-sha>, then diff the two versions as in step 4c. Cheaper than a patch per commit once the count climbs. Where the source's path is a directory, get_file_contents on it returns a listing rather than content — fetch each name in the entry's files list instead, at both refs, after narrowing that list per step 4.
   • Exception — changelog shape — a changelog appends at the top, so every per-commit patch is a small block of new lines while a naive two-full-file read is the entire file twice. At or below the ">5" count, read per-commit patches. Above it, the two-ref path is acceptable only when both refs are diffed on disk and only the new region enters context — what the exemption guards against is the file landing in the conversation, not the bytes being downloaded. Never let two full files into context to learn what an append-at-top diff already says. Measured on claude-code's ~590 KB CHANGELOG.md: 29 commits in a 35-day window were cheaper as patches; 85 commits across 89 days were cheaper as one local two-ref diff — 2 calls against 85. The break-even lies between the two and is not pinned down, so prefer the local diff once the count clears ">5" on a large file. Name the path taken in the report.
4. Size the fetch before you make it. Call count is a poor proxy for context cost — eight get_file_contents calls is cheap in calls and ~190 KB of markdown in the window. Two moves, in order:
   • Narrow by blob SHA. On a directory path, list it at both refs with fields: ["name", "size", "sha"]. That sha is the file's blob hash, so a name carrying the same sha at both refs did not change and must not be fetched; the two size values are the exact byte cost of fetching one that did. Two calls, no content, and it is more precise than a per-file list_commits sweep.
   • Spend a budget, in delta order. A changed file costs size(prior) + size(head) to diff. Budget roughly 150 KB per source per run, take the changed files in descending size-delta order, and stop when the budget is gone — naming any file left undiffed in the report. A source that cannot fit its own changed set is a signal its files list is too broad.
5. Strategy escape hatch. Commit count is only a proxy for patch size, and it breaks on repos that squash release branches. If a listing or a patch comes back oversized — truncated, worth spilling to a file, or plainly bigger than the alternative — switch strategy regardless of what the count said: from per-commit patches to a two-ref file diff, or via get_commit with detail: "stats" to see which files actually moved before pulling any patch at all. Name the strategy you ended on in the report.
6. Skip 4b entirely.

4b. Fallback path — WebFetch

Used when github-mcp is unavailable. Keeps the skill working for anyone cherry-picking it into a plain ~/.claude/.

• Raw markdown at any ref: https://raw.githubusercontent.com/<repo>/<sha-or-branch>/<path> — one URL per name in files when path is a directory
• Commits list: https://api.github.com/repos/<repo>/commits?path=<path>&per_page=<n> with optional &sha=<ref> or &since=<ISO-date>

GitHub anonymous API limit is 60 requests/hour and this path is unauthenticated. Price the selected set before running it, and price it in calls rather than sources. A single-file source costs ~3 (commits list, raw at prior ref, raw at HEAD) plus one re-fetch per sections entry when the completeness check below trips; a directory path costs 1 + 2 x |files| instead. Source count is therefore a bad proxy — one directory source with a long files list outweighs three single-file ones, and this line previously read "the four registered sources fit; a registry past roughly six does not," which was wrong on both the count and the driver.

Worked once, at 2026-09-08, against the registered set: fabric up to 14 (3 + its 11 named sections and workload subsections), vscode-agent 9 (4 files), fabric-iq-ontology 13 (6 files), powerbi ~0 GitHub calls — its entry override degrades to the Learn page, not to raw URLs — and claude-code 0, not attempted. That is ~36 of 60 worst case, so the set fits with less headroom than a per-source budget suggests. Re-derive it rather than trusting that figure. Once a run's projected calls approach the limit, the github-mcp path stops being optional.

A source the registry marks github-mcp-only — claude-code today — is not attempted on this path at all: name it in the report's skipped list rather than diffing a summary of it. A source whose entry defines its own fetch strategy is priced by that strategy and not by this rule.

Resolving the diff base — one rule, and it governs both fetch paths. If the user supplied a SHA whose repo matches this source, use that SHA directly. If a SHA from another source's repo, fall back to the date resolution. For a date, the base is the last commit that touched the source's path before the floor — list_commits with that path, until: <floor-date>T00:00:00Z, perPage: 1, fields: ["sha"].

Not "the parent of the oldest in-window commit", which is what this step said until 2026-09-07 and is wrong on any repo that merges release branches. The oldest in-window commit is typically the monthly release merge, so its first parent is a same-day main-branch commit that never touched the audited file and is dated inside the window. Measured on powerbi that day: the old rule resolved f1f53694 (2026-08-20, touching service-get-started.md and power-bi-visualization-card.md — not whats-new.md), where the correct rule resolves 443eb78f (2026-07-22, the July release merge, which is genuinely the file's previous version).

It survived undetected because it produced an identical diff — nothing else touched that file in between, so the two bases are byte-identical for path. That is luck, not correctness. Had any in-window commit touched the audited file before the release merge, its change would sit in the base and vanish from the diff: a silent under-report with no error path and no symptom in the output. Resolve the base by path, always.

Completeness check — re-fetch by section if summarized. WebFetch passes responses through a small LLM that summarizes pages above ~30–40 KB. Both registered What's New sources are ~50 KB; a broad "return the full markdown" prompt typically returns a bulleted feature list with table rows collapsed and descriptions dropped. After each fetch, scan the payload for tell-tale summarization: bullet lists where pipe-delimited tables should be, "additional sections truncated" language, missing description columns, or entry counts that look thin for a monthly cadence. If summarized, re-fetch the same URL with targeted section prompts — one per heading in the source's registry sections list, each asking for that section's content verbatim. Cache results in working memory keyed by section heading. Targeted prompts reliably return full content; broad ones do not. Skip this step only if the broad fetch returned complete, unsummarized content for every section.

A source whose registry entry has no sections list has no targeted-re-fetch escape hatch. If its broad fetch comes back summarized, report the source as fetch incomplete — github-mcp required rather than diffing a summary.

4c. Extract entries — by the source's shape

Walk prior and HEAD, splitting into entries per the registry's Shape contracts (table — rows keyed by the feature column; prose — heading blocks; changelog — version sections). For each entry present in HEAD but not in prior, capture:

• The section heading (and parent heading if relevant), or the version string for changelog.
• The entry's own fields — for table, the cells named by the entry's columns mapping; for prose and changelog, the entry text.
• Any link into the source's drill.host. Strip the anchor patterns named in drill.strip before storing — unstable anchors break across page revisions and pollute downstream diffing.
• Code or syntax examples — T-SQL, Spark SQL, KQL, DAX, M, TMDL, REST endpoints, CLI flags.
• MCP-related entries: new MCP servers, new tools on existing servers, transport / URL / authentication changes.
• Harness entries: hook events and their JSON fields, settings.json keys, frontmatter fields on skills / subagents / rules, ~/.claude/ path changes, permissions rule syntax, plugin and marketplace layout.

Apply the source's filter first if it has one. A filtered-out entry is bucket (d) — record it as a count, not a bullet, and never drill it.

Also capture removed entries as a separate set — a removal usually signals GA promotion (status cleared, row moved between tables) or a deprecation. Flag both kinds; preview-to-GA is high-value drift signal because skills often hedge on preview status.

Hold the diff in working memory; do not write it to disk.

5. Phase 2 — Map to current artifacts

For each diff entry, decide its bucket. Scan only the artifact classes the source's registry artifacts field lists:

• Skill match — Glob ~/.claude/skills/*/SKILL.md for directory-name keyword hits; Grep skill descriptions and bodies for feature-name and syntax keywords.
• Rule match — Grep ~/.claude/rules/coding-*.md for per-language overlap (a new T-SQL keyword lands on coding-tsql.md, a new DAX function on coding-dax.md, etc.).
• CLAUDE.md match — Read ~/.claude/CLAUDE.md; check whether a current instruction line is invalidated or extended by the entry.
• MCP match — Read ~/.claude/mcp/.mcp.global.template.json and ~/.claude/mcp/.mcp.project.template.json for the current server inventory across both global and per-project scopes; identify whether the entry adds, removes, or changes a server. Placing a finding is a two-step test. First, does it work from Claude Code at all? The Fabric-hosted api.fabric.microsoft.com/v1/mcp/* endpoints do not (OAuth DCR unsupported) and belong only in the VS Code workspace template, .vscode/mcp.template.json in the agent-config repo. Second, if it does work: is it bound to a workload (needs a workspace ID, database, connection string, or a running desktop app) or not? Workload-bound goes in the project template; cross-workload — docs, source control, cloud control plane — goes in the global one. Transport is not the test; a stdio server can be workload-bound and an http server can be universal.
• Harness match — Read ~/.claude/settings.json for the hook event wiring, permissions, and env keys currently in use; Glob ~/.claude/hooks/* and Grep them for event names and hook JSON fields; Grep skill / subagent / rule frontmatter for the fields the entry touches (allowed-tools, paths, model, argument-hint). A renamed hook event or a moved ~/.claude path breaks the deployed payload silently — there is no error path, so this scan is the only detection. Findings here land on the repo files the source's artifacts names, including scripts/link-claude.ps1 when a deployed location moves.

Classify each diff entry into exactly one bucket:

• (a) Drift / gap — an existing skill, rule, or CLAUDE.md line covers the topic and the entry changes the picture (new syntax, new limit, deprecation, behavioural change, GA-from-preview where the skill flagged preview status).
• (b) New-skill candidate — no current artifact covers the topic at all.
• (c) Tooling / MCP / CLI — affects an MCP template, CLAUDE.md tooling notes, or a referenced CLI's scope.
• (d) No-op — cosmetic, marketing, or unrelated to the current artifact scope.

If a diff entry straddles two buckets, split it on the way in. Forcing a single bucket per finding keeps the report actionable — each bullet maps to one follow-up task.

Do not bulk-rewrite from a single audit — drift edits are per-artifact decisions. A monthly diff can list ten flagged entries; that's ten separate per-skill rewrite tasks, not one batch.

6. Phase 3 — Selective upstream drill

Drilling upstream docs is expensive (token cost, latency, and brittle pages). Do it only where the answer affects an existing artifact's accuracy or a tooling decision. Open each link with the mechanism its source's drill.via names — microsoft-learn-mcp (microsoft_docs_fetch, which returns the full page rather than a summary) for the Microsoft sources, webfetch otherwise. Never assume a doc host; read it from the entry.

• Bucket (a) only — drill each linked page. Extract: syntax additions, new limits / quotas / retention windows, deprecations, behavioural changes that contradict current artifact content. Cite the specific URL and the specific change in the report.
• Bucket (b) — diff entry only; do not drill. New-skill scoping is its own task and over-drilling here pre-commits to authoring before the user has agreed the skill is worth writing.
• Bucket (c) — drill the linked MCP reference page if present; extract server name, transport, URL. Do not invent MCP transports or URLs. If an endpoint is announced without a clear reference page, mark it endpoint TBD — verify before template add and stop.
• Bucket (d) — skipped.

Graceful handling: paywall / login-wall / 404 / redirect-loop → note inline in the report (drill failed: <reason>) and continue. One broken link must not block the rest of the audit.

7. Phase 4 — Report format

Emit one markdown report to the conversation, sections in this exact order. If a bucket is empty, still include the heading with _(none)_ underneath — an empty heading is signal, not noise.

## Audit window

- Floor: <ISO date> (resolved from: <sha | date | default-35d>)
- Fetch path: <github-mcp | WebFetch fallback>
- Sources audited: <id, id> — skipped: <id, id | none>
- <label>: <commit-count> commits in window — prior `<prior-sha-or-floor>` (<prior-date>) → head `<head-sha>` (<head-date>)
  - `<sha>` (<date>) — <what the commit is>, `<path>` <+n/-m>
  - ...one line per in-window commit, newest first...
- ...one block per audited source, registry order...

## Drift / gap candidates (existing artifacts)

- **<artifact-slug>** — <diff entry feature name> _(<source-id>)_
  - Specific change: <what's new or different>
  - Reference: <anchor-stripped URL>
  - Proposed action: <flag | minor edit | partial rewrite>

## New-skill candidates

- **<feature name>** — <one-line rationale: skill vs CLAUDE.md line vs ignore>
  - Source: <source-id> / <section heading>

## MCP / tooling / CLI additions

- **<server or CLI name>** — <what's new>
  - Reference: <URL or "endpoint TBD — verify">
  - Proposed action: <add to ~/.claude/mcp/.mcp.global.template.json | ~/.claude/mcp/.mcp.project.template.json | .vscode/mcp.template.json | CLAUDE.md note | flag>

## No-op

- <short bullet, no detail>
- <short bullet, no detail>

## Recommended actions

1. <ordered, flag-only — name the artifact and the action verb>
2. ...

## Next run

Pass one of these as the prior reference next time:

- <label> head: `<head-sha>` (<head-date>)
- ...one line per audited source...
- Or a single date: `<today's ISO date>`

A SHA from any registered source's repo, or any ISO date, is accepted.

Report rules:

• Flag-only. Do not propose specific rewritten text — that is the per-artifact follow-up task's job.
• One artifact per drift bullet. If two artifacts overlap on the same diff entry, write two bullets.
• Every drift bullet names its source id. With a registry, "which page said this" is no longer inferable from the finding.
• URLs are anchor-stripped per the source's drill.strip, per the Phase 1 invariant.
• No invention. MCP transports, URLs, server names — quote the upstream reference or say "TBD". Never fabricate.
• Enumerate the in-window commits, never just the count. One line each, with its date and what it is. A bare "3 commits in window" is unfalsifiable: it hides which commits were inspected and which were only listed, and it strands a later reader who wants to reproduce the diff. Twice now — 2026-09-07 on powerbi, both runs — the per-commit list has been the thing someone needed to check afterwards. Where a commit was listed but never fetched, say so on its line rather than describing it.
• Next-run footer always printed. Even when buckets are empty — the SHAs are the user's handoff to next month.

8. Closing constraints

After emitting the report, restate the read-only contract briefly so any follow-up turn in the same conversation lands cleanly:

Audit complete. Read-only contract: no skill / rule / CLAUDE.md / MCP-template edits this turn. To capture the recommended actions as handoff briefs on disk, run /drift-handoff now — it reads this report from the conversation, so it has to run before the context is lost. A later session then executes those briefs with /drift-update. To act on a finding directly instead, start a new request naming the specific artifact and the action.

Constraints to honour throughout the turn:

• Read-only. Edit / Write / MultiEdit / NotebookEdit are off-limits.
• Registry-driven. Every repo, path, section, doc host, and anchor pattern comes from references/sources.md. Nothing about a source is hardcoded here.
• Selective drill. Upstream fetches only for bucket (a) and bucket (c)-with-reference.
• Anchor stripping. Every URL emitted to the report has the source's drill.strip patterns removed.
• No invention. Quote the upstream reference for MCP transports / URLs / server names, or say "TBD". Never fabricate.
• Per-artifact follow-up. Each flagged drift is its own task; never bulk-rewrite from one audit.