Documentation Audit

Documentation Audit Skill

Purpose

Audit and repair markdown documentation in codebases to ensure documentation claims match actual code behavior while maintaining documentation structure and completeness.

Success Criteria

1. Accuracy: All documentation claims verified against code
2. Navigation: Clear structure with functioning cross-links and README files where needed
3. Currency: Obsolete documentation archived or removed
4. Completeness: All significant features documented

Scope

Included:

• Markdown documentation files (*.md)
• Text-based diagrams (Mermaid, PlantUML, Graphviz, ASCII)
• Documentation structure and navigation

Excluded:

• Code comments and language-specific doc comments (godoc, JSDoc, etc.)
• Image-based diagrams
• Point-in-time documents (docs/plans/*)

When to Use This Skill

Use this skill when:

• Auditing documentation accuracy after significant code changes
• Onboarding to a new codebase and discovering documentation gaps
• Preparing for releases to ensure documentation is current
• Maintaining documentation as part of regular code health practices

Architecture: Multi-Pass Progressive Refinement

The skill operates in four passes, each building on the previous.

Pass 0: Repository Indexing

Build a comprehensive index of the repository before verification begins.

Create TodoWrite tasks for:

• Extract symbol graph using symbolic code analysis tools
• Identify API contracts from canonical sources (OpenAPI, protobuf, GraphQL schemas, configuration files)
• Map documentation cross-links and navigation files
• Index ownership metadata and last modification dates

Activities:

1. Use mcp__serena__find_symbol to extract exported symbols from each package
2. Use Glob to find canonical sources: **/*.proto, **/openapi.yaml, **/*.graphql, config schemas
3. Use Grep to map documentation links (search for [.*](.*\.md) patterns)
4. Note: Keep this pass lightweight - index only what's needed for verification

Output: Notes on repository structure, canonical sources found, and documentation topology.

Pass 1: Discovery and Classification

Discover all markdown files and classify them by purpose and lifecycle.

Create TodoWrite tasks for:

• Find all markdown files recursively
• Classify documents by type and lifecycle
• Identify obsolete candidates for archival

Activities:

1. Use Glob with pattern **/*.md to find all documentation
2. Classify using multiple signals:
   • Path heuristics: docs/archive/*, files with dates in names
   • Commit recency: Use Bash with git log --format=%ci --max-count=1 -- <file> to check last update
   • Link graph centrality: Documents frequently linked from others are likely living docs
   • YAML frontmatter: Look for status: living|archival|planned
3. Apply decision rules:
   • docs/plans/* are point-in-time (never updated)
   • High recency + clear structure = living documentation
   • Archived paths + low recency = obsolete candidate

Output: Classified document inventory with assessment of which docs to audit vs archive.

Safety: Move obsolete documents to docs/archive/ rather than delete. Ask user before archiving documents with ambiguous classifications.

Pass 2: Claim Extraction and Investigation Planning

Extract verifiable claims from living documentation and plan verification strategy.

Create TodoWrite tasks for each document section being validated.

Activities:

1. For each living documentation file:
   • Use Read to load the document
   • Extract claims by type:
     • Behavioral: "Service retries three times"
     • Structural: "Class implements Interface"
     • API: "Endpoint returns JSON with schema X"
     • Configuration: "Setting defaults to value Y"
     • Usage: "Run command with flag --foo"
2. Record claim metadata: doc_path, line_range, type, referenced_symbols
3. Identify text-based diagrams and classify:
   • Normative: Must match code structure exactly (UML class diagrams)
   • Illustrative: Conceptual, verify broad relationships only
4. Analyze codebase structure for documentation gaps:
   • Packages/modules without documentation
   • Exported APIs without usage examples
   • Directories lacking README files
5. Build investigation plan grouping claims by symbol/module for batch efficiency

Output: List of claims to verify, organized by module/package for efficient verification.

Optimization: Group related claims together to share context during verification.

Pass 3: Verification and Investigation

Verify claims against code using risk-appropriate methods.

Create TodoWrite tasks for each investigation batch.

Verification Hierarchy (use highest confidence method available):

1. Canonical sources: Verify against OpenAPI specs, protobuf definitions, GraphQL schemas, configuration schemas
2. Symbolic analysis: Verify structural claims using mcp__serena__find_symbol, mcp__serena__find_referencing_symbols, mcp__serena__search_for_pattern
3. Deep investigation: Use mcp__zen__analyze, mcp__zen__debug, or mcp__zen__thinkdeep for complex behavioral claims

Activities:

1. Process investigation batches from Pass 2
2. For each claim:
   • Select verification method based on claim type and available evidence
   • Start with symbolic analysis to narrow scope before deep investigation
   • Document findings with evidence trails (file paths, line numbers, symbol names)
   • Record whether claim is verified, contradicted, or requires user review
3. Verify diagrams:
   • Parse diagram syntax to extract assertions about code structure
   • Compare against symbol graph (normative) or verify broad structure (illustrative)
4. Create documentation for identified gaps:
   • Missing README files in documented directories
   • Undocumented features found in Pass 0 indexing
5. Handle conflicts:
   • Apply authority hierarchy: canonical sources > generated docs > top-level README > service READMEs
   • Record conflicts with context (version, environment, feature flags)
   • Annotate rather than delete when conflicts may be contextual

Output: Verified claims with corrections noted, new documentation drafts for gaps, conflict records.

Risk Management:

• Assign confidence to all verifications (exploring, low, medium, high, very_high, almost_certain, certain)
• Flag low-confidence verifications for user review
• Prefer static verification over runtime checks (safer, more deterministic)
• Never auto-fix based on weak evidence

Pass 4: Risk-Tiered Repair and Reporting

Apply corrections based on risk assessment.

Risk Tiers:

Auto-fix (apply immediately):

• Broken internal links (update paths)
• Typos in code references (symbol renamed in codebase)
• Outdated paths (files moved)
• Missing table of contents
• Diagram syntax errors

User approval required (present for review):

• Substantive technical corrections (behavior claims)
• Claim deletions or rewrites
• Document reclassifications
• Structural changes
• Conflict resolutions

User review required (flag but don't auto-fix):

• Low-confidence verifications
• Conflicting claims across documents
• Planned features (add disclaimers, don't verify behavior)
• Destructive actions (archival, removal)

Activities:

1. Apply auto-fixes immediately using Edit or Write tools
2. For user-approval changes:
   • Present diff preview
   • Show evidence trails (code spans, symbols verified, canonical sources)
   • Include confidence scores and rationale
   • Wait for user approval before applying
3. Move obsolete documents to docs/archive/ (after approval)
4. Create new README files for undocumented directories
5. Insert gap documentation using Write or Edit
6. Update navigation indexes
7. Generate summary report:
   • Changes by category and risk tier
   • Coverage metrics (claims verified / total claims)
   • Remaining manual review items
   • Confidence distribution

Output: Applied fixes, pending changes for review, archived documents, comprehensive summary.

Authority Hierarchy

When multiple documents make conflicting claims:

1. Canonical sources (OpenAPI, protobuf, schemas)
2. Generated documentation (godoc, JSDoc)
3. Top-level README.md
4. Service/module READMEs
5. Design documents
6. Ad-hoc notes

Document Classification Heuristics

Living Documentation (keep current):

• High commit frequency (updated within last 6 months)
• Clear ownership or maintenance signals
• Linked from README or navigation
• Path patterns: README.md, CONTRIBUTING.md, ARCHITECTURE.md, top-level docs

Point-in-Time (never update):

• Under docs/plans/
• Contains specific dates or version numbers
• Marked with status: plan or status: archival
• Low change frequency (<2 updates ever)

Obsolete (archive):

• No updates in 12+ months
• References removed code
• Superseded by newer documentation
• Marked with DRAFT or TODO

Skill Parameters

Accept these parameters when invoked:

• dry_run (default: true): Generate report without applying changes
• risk_tier (default: user_approval): Maximum risk tier to auto-apply (auto_fix | user_approval)
• verification_method (default: auto): Force specific verification approach (auto | canonical | symbolic | deep)
• focus_paths (default: all): Restrict to specific directories (e.g., "docs/", "README.md")
• skip_gap_analysis (default: false): Skip documentation gap detection

Usage Workflow

Step 1: Announce and Initialize

"I'm using the documentation-audit skill to verify documentation accuracy."

Create TodoWrite tasks for all four passes plus a completion task.

Step 2: Execute Pass 0 - Repository Indexing

Follow Pass 0 activities above. Keep this lightweight - just enough to understand repository structure and find canonical sources.

Step 3: Execute Pass 1 - Discovery and Classification

Follow Pass 1 activities above. Classify all markdown files and identify obsolete candidates.

Step 4: Execute Pass 2 - Claim Extraction

Follow Pass 2 activities above. Extract all verifiable claims and create investigation plan.

Step 5: Execute Pass 3 - Verification

Follow Pass 3 activities above. Verify each claim using appropriate verification hierarchy.

Step 6: Execute Pass 4 - Repair

Follow Pass 4 activities above. Apply fixes according to risk tiers and present changes for review.

Step 7: Summary

Provide conversational summary:

• Total claims verified
• Auto-fixes applied
• Changes requiring approval
• Items requiring manual review
• Overall documentation health assessment

Tools Available

• Serena symbolic tools: mcp__serena__find_symbol, mcp__serena__find_referencing_symbols, mcp__serena__search_for_pattern, mcp__serena__get_symbols_overview
• Zen investigation tools: mcp__zen__analyze, mcp__zen__debug, mcp__zen__thinkdeep, mcp__zen__codereview
• File operations: Read, Write, Edit, Glob, Grep
• Version control: Bash with git commands for commit history

Token Efficiency Strategies

1. Index once, reuse: Pass 0 creates repository knowledge for later passes
2. Batch similar claims: Group by symbol/module to share context
3. Retrieval-augmented: Fetch only relevant code spans for verification
4. Prefer static: Use symbolic analysis before deep investigation
5. Focused reading: Use mcp__serena__get_symbols_overview before reading full files

Safety Mechanisms

1. Evidence trails: Every change documents source claim, verification method, evidence
2. Confidence scoring: Low confidence blocks auto-fixes
3. Diff previews: Show changes before applying
4. Archival over deletion: Move obsolete docs, don't delete
5. Risk tiers: Separate auto-fixable issues from those requiring approval

Edge Cases

Feature flags: Claims may be contextual (true when flag enabled). Record context, don't mark as contradicted.

Multiple versions: Documentation for different versions may coexist. Partition claims by version.

Planned features: Add disclaimer banners, mark verification: not_applicable. Don't attempt behavioral verification.

External dependencies: Claims about external APIs may be stale. Verify against canonical sources if available, otherwise flag for review.

Runtime behavior: Claims requiring execution (performance, flakiness) are fragile. Prefer symbolic verification or mark for manual testing.

Example Execution

User: Run documentation audit on docs/ with auto-fixes enabled

Claude:
1. "I'm using the documentation-audit skill to verify documentation accuracy."
2. Creates TodoWrite for 4 passes + summary
3. Pass 0: Indexes repository (symbols, canonical sources, doc links)
4. Pass 1: Finds 23 markdown files, classifies 3 as obsolete candidates
5. Pass 2: Extracts 156 claims from 20 living docs, groups into 12 investigation batches
6. Pass 3: Verifies claims using symbolic analysis (78%), canonical sources (12%), deep investigation (10%)
   - Finds 8 contradicted claims, 12 documentation gaps
7. Pass 4:
   - Auto-fixes: 3 broken links, 1 outdated path
   - Presents for approval: 8 technical corrections, 12 new documentation sections
   - Flags for review: 2 low-confidence verifications
8. Summary: "Verified 156 claims with 95% confidence. Applied 4 auto-fixes, 20 changes await approval, 2 require manual review."

Anti-Patterns

❌ Don't skip symbolic analysis - Use serena tools before reading full files ❌ Don't auto-fix low-confidence findings - Flag for user review instead ❌ Don't delete documents - Archive to docs/archive/ instead ❌ Don't batch all changes - Apply auto-fixes immediately, present substantive changes for review ❌ Don't verify point-in-time documents - Leave docs/plans/* alone ❌ Don't skip TodoWrite - Track progress through all four passes

Summary

This skill performs rigorous documentation audits through four systematic passes:

1. Index repository structure and canonical sources
2. Classify documents by lifecycle and purpose
3. Extract and group verifiable claims
4. Verify claims and apply risk-tiered repairs

90% of effort is verification against code, 10% is structure and quality.