Documentation Audit

Documentation Audit Skill

Systematically verify claims in documentation against the actual codebase using a two-pass approach.

When This Skill Activates

• Auditing documentation for accuracy
• Before release cycles (verify docs match current behavior)
• After major refactors (code changed, docs may be stale)
• When users report docs don't match behavior
• Periodic hygiene (quarterly documentation review)

Trigger phrases: "audit docs", "verify documentation", "check docs", "docs accurate", "documentation drift"

Core Principle

Low recall is worse than false positives — missed claims stay invisible and mislead users.

Two-pass process:

1. Pass 1: Extract and verify claims directly from docs
2. Pass 2A: Expand patterns from false claims to find similar issues
3. Pass 2B: Compare codebase inventory vs documented items (gap detection)

Quick Start

1. Identify target docs (user-facing only, skip plans/, audits/)
2. Note current git commit for report header
3. Run Pass 1 extraction using parallel agents (one per doc)
4. Analyze false claims for patterns
5. Run Pass 2 expansion searches
6. Generate docs/audits/AUDIT_REPORT_YYYY-MM-DD.md

Claim Types

Verification Confidence Tiers

• Tier 1 (auto): file_ref, config_default, env_var, cli_command
• Tier 2 (semi-auto): symbol_ref, version_req
• Tier 3 (human review): behavior, constraint

Pass 2 Pattern Expansion

After Pass 1, analyze false claims and search for similar patterns:

Dead script found: diagnose_track_selection.py
  → Search: all script references → Found 8 more dead scripts

Wrong interval: "every 10 seconds"
  → Search: "every \d+ (seconds?|minutes?)" → Found 3 more

Wrong service name: ai-radio-break-gen.service
  → Search: service/timer names → Found naming inconsistencies

Common Patterns to Check

Always verify these patterns across all docs:

• Dead scripts: scripts/*.py references
• Timer intervals: every \d+ (seconds?|minutes?)
• Service names: ai-radio-*.service, *.timer
• Config vars: RADIO_* environment variables
• CLI flags: --flag patterns in bash blocks

Output Format

Generate docs/audits/AUDIT_REPORT_YYYY-MM-DD.md:

# Documentation Audit Report
Generated: YYYY-MM-DD | Commit: abc123

## Executive Summary
| Metric | Count |
|--------|-------|
| Documents scanned | 12 |
| Claims verified | ~180 |
| Verified TRUE | ~145 (81%) |
| **Verified FALSE** | **31 (17%)** |

## False Claims Requiring Fixes
### CONFIGURATION.md
| Line | Claim | Reality | Fix |
|------|-------|---------|-----|
| 135 | `claude-sonnet-4-5` | Actual: `claude-3-5-sonnet-latest` | Update |

## Pattern Summary
| Pattern | Count | Root Cause |
|---------|-------|------------|
| Dead scripts | 9 | Scripts deleted, docs not updated |

## Human Review Queue
- [ ] Line 436: behavior claim needs verification

Execution Approach

This skill runs in Plan mode with a forked context to keep extraction artifacts separate from your main conversation:

1. Parallel extraction - Use one Task agent per document for efficiency
2. Evidence-based verdicts - Record file:line for every claim verification
3. Pattern matching - After Pass 1, systematically search for similar issues
4. Gap detection - List actual artifacts vs documented ones

Anti-Patterns to Avoid

1. Skipping Pass 2 — Pattern expansion catches 10-20% more issues
2. Trusting "looks correct" — Always verify with evidence
3. Fixing without evidence — Always cite file:line or "not found"
4. Auditing design docs — Focus on user-facing docs, skip historical artifacts
5. Batching verification early — Complete extraction first

Decision Framework

What to Audit

When to Run

Real-World Results

From a representative audit:

• 12 documents scanned
• ~180 claims verified
• 31 false claims found (17% error rate)
• Common patterns: dead scripts (9), wrong intervals (4), wrong service names (3)

Detailed References

For comprehensive coverage of specific topics, see:

• references/checklist.md — Step-by-step execution checklist with TodoWrite integration
• references/extraction-patterns.md — Detailed patterns for claim recognition and verification