Doc Accuracy

Documentation Accuracy Skill

Verify documentation claims against actual code behavior. Code is truth; docs are the subject under test.

Triggers

When to Use

Use this skill when:

• Documentation may contain non-compilable code examples
• Behavioral claims in docs may contradict implementation
• Quantitative claims (performance, limits) appear in multiple files with different values
• Preparing for a release and need a documentation accuracy audit
• API reference may be missing public members

Use direct code review instead when:

• Investigating a single known documentation error
• The inaccuracy is already identified and needs a fix

Replaces

Architecture

Asymmetric Verification

Code compiles and runs. Documentation describes what code does. When they disagree, the code is right. This skill reads code first, builds a verified model, then checks documentation claims against that model.

Phase Overview

Phase 1: Assessment        (script-only, <30s)  -> assessment.json
Phase 2: Claim Extraction  (script-only, <15s)  -> claims.json
Phase 3: Compilability     (script-only, <60s)  -> compilability-findings.json
Phase 4: Behavioral        (Sonnet agents, 3-7m) -> behavioral-findings.json
Phase 5: Cross-Document    (script + Sonnet, 1-2m) -> consistency-findings.json
Phase 6: Structure         (Sonnet agent, 1-2m)  -> structure-findings.json

Scripts

Invocation

# Full deterministic scan (Phases 1-3)
python3 scripts/doc_accuracy.py --target /path/to/repo

# Compilability only
python3 scripts/doc_accuracy.py --target /path/to/repo --phases 3

# Incremental (changed files only)
python3 scripts/doc_accuracy.py --target /path/to/repo --diff-base main

# JSON output to specific directory
python3 scripts/doc_accuracy.py --target /path/to/repo --output-dir .doc-accuracy

# Set severity threshold for exit code
python3 scripts/doc_accuracy.py --target /path/to/repo --severity-threshold critical

# Markdown report output
python3 scripts/doc_accuracy.py --target /path/to/repo --format markdown

# Text summary to stdout
python3 scripts/doc_accuracy.py --target /path/to/repo --format summary

Output Artifacts

Process

Phases 1-3: Deterministic (Script)

Run doc_accuracy.py to produce JSON artifacts. No LLM calls.

1. Phase 1 (Assessment): Enumerate documentation and source files. Extract public symbols via regex. Build doc-to-source mapping.
2. Phase 2 (Claim Extraction): Parse markdown files. Extract verifiable claims with file path, line number, claim type, and referenced symbols.
3. Phase 3 (Compilability): Verify type names, method names, parameter names in code examples exist in the codebase via symbol index lookup. If Phase 1 finds no source symbols, report DID_NOT_RUN with zero findings instead of treating every documentation reference as unresolved.

Phase 4: Behavioral Verification (Agent)

Dispatch one Sonnet agent per file group. Each agent receives:

• Full source file content (read code FIRST)
• Documentation file content
• Filtered claims from claims.json
• Compilability findings (to avoid re-checking)

Agent prompt:

You are verifying documentation accuracy. Code is the source of truth.

SOURCE FILES (read these first):
[full content of mapped source files]

DOCUMENTATION FILE:
[full content of the documentation file]

CLAIMS TO VERIFY:
[filtered claims from claims.json]

For each claim:
1. Find the relevant code in the source files
2. Determine if the documentation claim accurately describes the code behavior
3. If inaccurate: severity, description, evidence (with line numbers), suggested fix
4. If accurate: mark as PASS

Additionally check for:
- Public API members in source absent from documentation
- Behavioral nuances the documentation omits or misrepresents
- Default values that differ between docs and code

Launch agents in parallel (one per file group).

Phase 5: Cross-Document Consistency (Agent)

From claims.json, filter to quantitative and behavioral claims. Group by topic. For groups with conflicting values across files, dispatch a Sonnet agent with benchmark data to determine the correct value.

Phase 6: Structure and Quality (Agent)

Validate documentation structure (indexes, navigation, completeness). Apply comment quality framework (accuracy, completeness, long-term value, misleading elements, improvements) to a 20% sample of source comments.

Reconciliation (Interactive)

Present proposed fixes for user approval before modifying any file. Categories:

• Documentation prose fixes
• Code example corrections
• Consistency resolution (single source of truth)
• Structural updates (indexes, navigation)

Issue Taxonomy

Severity Levels

Exit Codes

Canonical source bundled with this skill:

scripts/doc_accuracy.py

Exit Codes:
    0: No findings at or above severity threshold
    1: Error or inconclusive run, including no source symbols for Phase 3
    2: Configuration error, including an invalid --diff-base
    3: External dependency failure, including unavailable or failed Git
    10: Findings at or above severity threshold

Anti-Patterns

Verification

After running:

• All JSON artifacts created in output directory
• Every finding has file path, line number, severity, and evidence
• Code examples verified against actual method signatures
• Behavioral claims verified by reading implementation source
• Cross-document conflicts identified with all locations listed
• Exit code reflects severity threshold
• Phase 3 reports DID_NOT_RUN when no source symbols exist

Related Skills