Doc Auditor

Documentation Auditor Skill

This skill is used for:

• Initial audit: Analyzing existing documentation for fractary-docs setup
• Ongoing compliance: Checking managed docs against evolving standards
• Quality assurance: Regular documentation health checks

You generate specifications that can be followed to bring documentation into compliance.

IMPORTANT: Use fractary-spec Plugin When Available

• Check if fractary-spec plugin is installed
• If available: Use spec-manager agent (fractary-spec:spec-manager) to generate standardized spec
• If not available: Cannot generate spec (audit presents findings only)
• Either way, output must be actionable

IMPORTANT: Respect Project-Specific Documentation

• Only audit documentation types the plugin has opinions on
• Leave project-specific docs outside plugin scope alone
• Identify and preserve unique project requirements

IMPORTANT: Two-Phase Interactive Workflow with State Tracking

This skill executes in TWO phases with a mandatory user approval step:

Phase 1: Analysis & Presentation (Steps 1-6)

• Discover documentation state
• Analyze against standards
• Identify issues and remediation actions
• Present findings to user for review
• STOP and wait for approval
• State: AWAITING_USER_DECISION

Phase 2: Specification Generation (Steps 7-8)

• ONLY execute after explicit user approval
• State: USER_APPROVED → GENERATING_SPEC
• Create GitHub tracking issue
• Generate formal remediation specification via spec-manager agent
• Present final summary
• State: COMPLETED

Phase State Management:

• Track current phase state to prevent accidental skipping
• Never transition from Phase 1 to Phase 2 without user approval
• Valid state transitions:
  • ANALYZING → AWAITING_USER_DECISION (end of Step 6)
  • AWAITING_USER_DECISION → USER_APPROVED (user chooses "Save as Spec")
  • AWAITING_USER_DECISION → REFINING (user chooses "Refine Plan")
  • AWAITING_USER_DECISION → CANCELLED (user chooses "Hold Off")
  • USER_APPROVED → GENERATING_SPEC (Step 7 begins)
  • GENERATING_SPEC → COMPLETED (Step 8 done)

CRITICAL: Never skip the approval step in Step 6. Always present findings first and wait for user to approve, revise, or cancel.

Step 1: Check for Spec Plugin

CRITICAL: The fractary-spec plugin is REQUIRED for generating remediation specs.

Execute plugin availability check and store state for later steps:

#!/bin/bash

# Set up directories with timestamp
timestamp=$(date +%Y-%m-%d-%H%M%S)

# Temporary discovery files go in tmp directory
temp_dir="${temp_dir:-logs/audits/tmp}"
mkdir -p "$temp_dir"

# Final audit report will be saved in logs/audits/docs with timestamp
audit_report_path="${audit_report_path:-logs/audits/docs/$timestamp-documentation-audit.md}"
mkdir -p "logs/audits/docs"

# State directory for workflow state (separate from logs)
state_dir=".fractary/state"
mkdir -p "$state_dir"

echo "Temporary discovery files: $temp_dir"
echo "Final audit report: $audit_report_path"

# Check for spec plugin in both local project and global plugin directory
SPEC_PLUGIN_AVAILABLE=false

if [ -f ".fractary/plugins/spec/config.json" ]; then
  echo "✓ Found spec plugin in project config"
  SPEC_PLUGIN_AVAILABLE=true
elif [ -d "plugins/spec" ] && [ -f "plugins/spec/.claude-plugin/plugin.json" ]; then
  echo "✓ Found spec plugin in plugins directory"
  SPEC_PLUGIN_AVAILABLE=true
else
  echo "⚠️  WARNING: fractary-spec plugin not found"
  echo "   Searched:"
  echo "   - .fractary/plugins/spec/config.json"
  echo "   - plugins/spec/.claude-plugin/plugin.json"
  echo ""
  echo "The audit will present findings, but cannot generate a formal spec."
  echo "To enable spec generation, install fractary-spec plugin."
  echo ""
  SPEC_PLUGIN_AVAILABLE=false
fi

# Store plugin availability state in state directory (not in logs)
# Using file-based state tracking for persistence across LLM turns
if [ "$SPEC_PLUGIN_AVAILABLE" = "true" ]; then
  echo "available" > "$state_dir/audit-spec-plugin-status.txt"
else
  echo "unavailable" > "$state_dir/audit-spec-plugin-status.txt"
fi

# Store paths for use in later steps
echo "$temp_dir" > "$state_dir/audit-temp-dir.txt"
echo "$audit_report_path" > "$state_dir/audit-report-path.txt"
echo "$timestamp" > "$state_dir/audit-timestamp.txt"

echo "State saved to: $state_dir/"

Directory Structure:

• Temporary discovery files: logs/audits/tmp/ (deleted after final report generated)
• Final audit reports: logs/audits/docs/{timestamp}-documentation-audit.md (permanent, tracked by logs-manager)
• Workflow state: .fractary/state/ (ephemeral workflow state)

State Tracking:

• Plugin availability is stored in .fractary/state/audit-spec-plugin-status.txt
• Paths stored in .fractary/state/ for access across workflow steps
• File contains either "available" or "unavailable"
• This persists across workflow steps and LLM turns

Behavior based on plugin availability:

If plugin status is "available":

• Continue with full workflow (Steps 2-8)
• User will see "Save as Spec" option in Step 6
• Can generate formal specification via spec-manager agent (fractary-spec:spec-manager)

If plugin status is "unavailable":

• Continue with audit and present findings (Steps 2-6 only)
• User will see modified options in Step 6 (no "Save as Spec")
• User can still:
  • Review detailed findings
  • Refine the remediation plan
  • Access discovery reports
• Workflow ends after Step 6 unless plugin is installed

Step 2: Load Configuration and Standards

Load fractary-docs configuration (if exists):

• Project config: .fractary/plugins/docs/config.json
• Plugin defaults: plugins/docs/config/config.example.json

Load project-specific standards (if configured):

• Check config for validation.project_standards_doc
• Read project standards document

Step 3: Discover Documentation State

Execute discovery scripts and write results to temporary directory:

# Read temp directory from state
temp_dir=$(cat .fractary/state/audit-temp-dir.txt)

# Execute discovery scripts
bash plugins/docs/skills/doc-auditor/scripts/discover-docs.sh {project_root} $temp_dir/discovery-docs.json
bash plugins/docs/skills/doc-auditor/scripts/discover-structure.sh {project_root} $temp_dir/discovery-structure.json
bash plugins/docs/skills/doc-auditor/scripts/discover-frontmatter.sh $temp_dir/discovery-docs.json $temp_dir/discovery-frontmatter.json
bash plugins/docs/skills/doc-auditor/scripts/assess-quality.sh $temp_dir/discovery-docs.json $temp_dir/discovery-quality.json

Temporary Files: These discovery reports are temporary working files in logs/audits/tmp/ that will be cleaned up after the final audit report is generated.

Step 4: Analyze Against Standards

Load discovery results and compare against standards:

Plugin Standards (Always Applied):

• Front matter requirements (title, type, status, date, tags, codex_sync)
• File organization (ADRs in architecture/adrs/, designs in architecture/designs/, etc.)
• Required sections per doc type (from config validation.required_sections)
• Naming conventions (ADR-NNN-title.md, etc.)

Project Standards (If Configured):

• Custom validation rules from project_standards_doc
• Custom hooks and validation scripts
• Project-specific naming or organization

Identify Issues:

• Missing front matter
• Files in wrong locations
• Missing required sections
• Broken links
• Quality issues (incomplete docs, poor structure)
• Non-compliant naming

Step 5: Generate Remediation Actions

For each issue identified, create remediation action:

Action Types:

• add-frontmatter: Add/update front matter
• move-file: Relocate file to standard location
• add-section: Add missing required section
• fix-link: Fix broken cross-reference
• improve-quality: Add structure, content, examples
• rename-file: Align with naming conventions

Prioritization:

• HIGH: Blocks codex sync, validation, or core functionality
• MEDIUM: Organization, structure, best practices
• LOW: Nice-to-haves, optimizations

Step 6: Present Findings to User

CRITICAL: This is an interactive approval step.

Check plugin availability state and present the audit findings to the user for review.

Read plugin status from state file:

state_dir=".fractary/state"
if [ -f "$state_dir/audit-spec-plugin-status.txt" ]; then
  PLUGIN_STATUS=$(cat "$state_dir/audit-spec-plugin-status.txt")
else
  PLUGIN_STATUS="unavailable"
fi

Present the audit findings and proposed remediation actions:

═══════════════════════════════════════════════════════════
📊 DOCUMENTATION AUDIT FINDINGS
═══════════════════════════════════════════════════════════

📄 DOCUMENTATION INVENTORY
  Total Files: {count}
  By Type: ADRs: {n}, Designs: {n}, Runbooks: {n}, Other: {n}

📊 COMPLIANCE STATUS
  Front Matter Coverage: {percentage}% ({with}/{total})
  Quality Score: {score}/10
  Organization: {status}

⚠️ ISSUES IDENTIFIED
  High Priority: {count}
  Medium Priority: {count}
  Low Priority: {count}

📋 PROPOSED REMEDIATION ACTIONS

### High Priority ({count} actions)
1. [Action description with affected files]
2. [Action description with affected files]
...

### Medium Priority ({count} actions)
1. [Action description with affected files]
...

### Low Priority ({count} actions)
1. [Action description with affected files]
...

⏱️ ESTIMATED EFFORT
  High Priority: {hours} hours
  Medium Priority: {hours} hours
  Low Priority: {hours} hours
  Total: {total_hours} hours

═══════════════════════════════════════════════════════════

📋 What would you like to do next?

{IF $PLUGIN_STATUS == "available":}
1. **Save as Spec**: Generate a formal remediation specification using
   fractary-spec:spec-manager (recommended for tracking and execution)

2. **Refine Plan**: Provide feedback to adjust priorities, actions, or scope
   (I'll revise and present an updated plan for your approval)

3. **Hold Off**: Save these findings for later without generating a spec
   (Discovery reports remain available for future reference)

{IF $PLUGIN_STATUS == "unavailable":}
⚠️  Note: fractary-spec plugin not installed - cannot generate formal spec

1. **Refine Plan**: Provide feedback to adjust priorities, actions, or scope
   (I'll revise and present an updated plan for your approval)

2. **Hold Off**: Save these findings for later
   (Discovery reports remain available for future reference)

   To enable spec generation, install fractary-spec plugin first.

═══════════════════════════════════════════════════════════

STOP HERE and wait for user response.

User Decision Handling:

• "Save as Spec" (or similar approval) → Proceed to Step 7
• "Refine Plan" (or requests changes) → Revise actions and re-present Step 6
• "Hold Off" (or cancels) → Skip to completion with discovery reports only

Do NOT proceed to spec generation until user explicitly chooses "Save as Spec".

Step 7: Generate Remediation Specification (After "Save as Spec")

ONLY execute this step if user explicitly chooses to save as spec in Step 6.

CRITICAL: The spec-manager agent requires an issue number for tracking. Create a GitHub issue first.

Step 7.1: Create Tracking Issue

Use the @agent-fractary-work:work-manager agent to create a GitHub issue for tracking the remediation work.

Natural Language Invocation:

Use the @agent-fractary-work:work-manager agent to create a GitHub issue with the following request:

{
  "operation": "create-issue",
  "parameters": {
    "title": "Documentation Remediation - {project_name}",
    "body": "Audit identified {total_issues} compliance issues with documentation:\n\n**High Priority**: {high_count} issues\n**Medium Priority**: {medium_count} issues\n**Low Priority**: {low_count} issues\n\n**Quality Score**: {score}/10\n**Compliance**: {percentage}%\n**Estimated Effort**: {hours} hours\n\nA detailed remediation specification will be generated to address these issues.",
    "labels": ["documentation", "remediation", "automated-audit"],
    "assignee": null
  }
}

Note: The JSON structure above specifies the request parameters for the agent. The agent is invoked via natural language declaration, and the system routes the request automatically.

Capture the returned issue number for use in Step 7.2.

Step 7.2: Generate Spec via Spec-Manager

Use the @agent-fractary-spec:spec-manager agent to generate a formal specification from the tracking issue.

Natural Language Invocation:

Use the @agent-fractary-spec:spec-manager agent to generate specification with the following request:

{
  "operation": "generate",
  "issue_number": "{issue_number_from_step_7.1}",
  "parameters": {
    "template": "infrastructure",
    "force": false
  }
}

Note: The JSON structure above specifies the request parameters for the agent. The agent is invoked via natural language declaration, and the system routes the request automatically.

The spec-manager agent will:

1. Fetch the issue created in Step 7.1
2. Use the issue body and audit findings to populate the spec
3. Generate spec file: specs/spec-{issue_number}-documentation-remediation.md (path configurable via spec plugin)
4. Link the spec back to the issue via GitHub comment
5. Return the spec path for verification

Step 7.3: Register Audit Logs

Use the @agent-fractary-logs:logs-manager agent to register the audit logs for tracking.

Natural Language Invocation:

Use the @agent-fractary-logs:logs-manager agent to register audit logs with the following request:

{
  "operation": "register-log",
  "parameters": {
    "log_type": "audit",
    "log_directory": "logs/audit/{timestamp}",
    "metadata": {
      "project_name": "{project_name}",
      "total_files": {count},
      "issues_found": {total_issues},
      "quality_score": {score},
      "spec_generated": true,
      "tracking_issue": "{issue_number}"
    }
  }
}

Note: The logs-manager tracks all audit runs over time, allowing you to see compliance trends and history.

Step 7.4: Generate Standardized Audit Report via docs-manage-audit

Invoke docs-manage-audit skill to create dual-format audit report:

Skill(skill="docs-manage-audit")

Then provide the audit data:

Use the docs-manage-audit skill to create documentation audit report with the following parameters:
{
  "operation": "create",
  "audit_type": "documentation",
  "check_type": "full",
  "audit_data": {
    "audit": {
      "type": "documentation",
      "check_type": "full",
      "timestamp": "{ISO8601}",
      "duration_seconds": {duration},
      "auditor": {
        "plugin": "fractary-docs",
        "skill": "doc-auditor"
      },
      "audit_id": "{timestamp}-documentation-audit"
    },
    "summary": {
      "overall_status": "pass|warning|error",
      "status_counts": {
        "passing": {passing_count},
        "warnings": {warning_count},
        "failures": {failure_count}
      },
      "exit_code": {0|1|2},
      "score": {quality_score},
      "compliance_percentage": {compliance_percentage}
    },
    "findings": {
      "categories": [
        {
          "name": "Front Matter",
          "status": "pass|warning|error",
          "checks_performed": {count},
          "passing": {count},
          "warnings": {count},
          "failures": {count}
        },
        {
          "name": "Structure",
          "status": "pass|warning|error",
          "checks_performed": {count},
          "passing": {count},
          "warnings": {count},
          "failures": {count}
        },
        {
          "name": "Quality",
          "status": "pass|warning|error",
          "checks_performed": {count},
          "passing": {count},
          "warnings": {count},
          "failures": {count}
        }
      ],
      "by_severity": {
        "high": [
          {
            "id": "doc-001",
            "severity": "high",
            "category": "frontmatter",
            "message": "{issue description}",
            "resource": "{file path}",
            "remediation": "{how to fix}"
          }
        ],
        "medium": [{finding}],
        "low": [{finding}]
      }
    },
    "metrics": {
      "documentation_count": {total_files},
      "coverage_percentage": {frontmatter_coverage}
    },
    "recommendations": [
      {
        "priority": "high|medium|low",
        "category": "documentation",
        "recommendation": "{action}",
        "effort_days": {estimated_effort}
      }
    ],
    "extensions": {
      "documentation": {
        "frontmatter_coverage": {frontmatter_coverage_percentage},
        "quality_score": {quality_score},
        "gap_categories": [{gap_category}],
        "remediation_spec_path": "specs/spec-{issue_number}-documentation-remediation.md",
        "tracking_issue_url": "{issue_url}"
      }
    }
  },
  "output_path": "logs/audits/docs/",
  "project_root": "{project-root}"
}

This will generate:

• README.md: Human-readable audit dashboard
• audit.json: Machine-readable audit data

Both files in logs/audits/docs/{timestamp}-documentation-audit.[md|json]

Cleanup temporary files:

# Read paths from state
temp_dir=$(cat .fractary/state/audit-temp-dir.txt)

# Clean up temporary discovery files
echo "Cleaning up temporary files from $temp_dir/"
rm -f "$temp_dir"/*

Integration: The docs-manage-audit skill standardizes the audit report format and integrates with logs-manager for retention tracking.

Step 8: Present Final Summary to User (After Spec Generation)

Display audit completion summary:

═══════════════════════════════════════════════════════════
📊 DOCUMENTATION AUDIT SUMMARY
═══════════════════════════════════════════════════════════

📄 DOCUMENTATION INVENTORY
  Total Files: {count}
  By Type: ADRs: {n}, Designs: {n}, Runbooks: {n}, Other: {n}

📊 COMPLIANCE STATUS
  Front Matter Coverage: {percentage}% ({with}/{total})
  Quality Score: {score}/10
  Organization: {status}

⚠️ ISSUES IDENTIFIED
  High Priority: {count}
  Medium Priority: {count}
  Low Priority: {count}

📋 OUTPUTS
  Audit Report: logs/audits/docs/{timestamp}-documentation-audit.md
  Remediation Spec: specs/spec-{issue_number}-documentation-remediation.md
  Tracking Issue: #{issue_number}
  Estimated Effort: {hours} hours

💡 NEXT STEPS
  1. Review audit report: logs/audits/docs/{timestamp}-documentation-audit.md
  2. Review remediation spec: specs/spec-{issue_number}-documentation-remediation.md
  3. Follow implementation plan
  4. Verify with: /fractary-docs:validate
  5. View audit history: logs/audits/docs/

═══════════════════════════════════════════════════════════

OUTPUT END MESSAGE:

✅ COMPLETED: Documentation Audit
Issues Found: {count}
Spec Generated: {path}
───────────────────────────────────────────────────────────
Next: Review and follow remediation spec

Phase 1 Complete When:

• All discovery scripts have executed
• Documentation analyzed against standards
• Remediation actions identified and prioritized
• Findings presented to user in structured format
• User prompted for approval/revision/cancellation
• Skill pauses and waits for user response

Phase 2 Complete When (After User Approval):

• GitHub tracking issue created
• Specification generated via spec-manager agent
• Spec file created at specs/spec-{issue_number}-documentation-remediation.md
• Final summary presented to user
• Next steps provided with spec path and tracking issue

If User Cancels:

• No spec is generated
• Findings remain available for reference
• Discovery reports saved for future use

Phase 1 Output (Findings Presentation):

{
  "success": true,
  "operation": "audit",
  "phase": "findings_presentation",
  "result": {
    "total_files": 23,
    "issues": {
      "high": 5,
      "medium": 7,
      "low": 3,
      "total": 15
    },
    "quality_score": 6.2,
    "compliance_percentage": 45,
    "estimated_hours": 8,
    "temp_discovery_dir": "logs/audits/tmp",
    "discovery_reports": [
      "logs/audits/tmp/discovery-docs.json",
      "logs/audits/tmp/discovery-structure.json",
      "logs/audits/tmp/discovery-frontmatter.json",
      "logs/audits/tmp/discovery-quality.json"
    ],
    "awaiting_user_approval": true
  },
  "timestamp": "2025-01-15T14:30:22Z"
}

Phase 2 Output (After Spec Generation via spec-manager):

{
  "success": true,
  "operation": "audit",
  "phase": "spec_generation_complete",
  "result": {
    "total_files": 23,
    "issues": {
      "high": 5,
      "medium": 7,
      "low": 3,
      "total": 15
    },
    "quality_score": 6.2,
    "compliance_percentage": 45,
    "audit_report_path": "logs/audits/docs/2025-01-15-143022-documentation-audit.md",
    "spec_path": "specs/spec-123-documentation-remediation.md",
    "tracking_issue_number": "123",
    "tracking_issue_url": "https://github.com/org/repo/issues/123",
    "estimated_hours": 8,
    "spec_manager_used": true,
    "logs_registered": true,
    "temp_files_cleaned": true
  },
  "timestamp": "2025-01-15T14:35:45Z"
}

Error Response:

{
  "success": false,
  "operation": "audit",
  "error": "No documentation files found",
  "error_code": "NO_DOCS_FOUND",
  "timestamp": "2025-01-15T12:00:00Z"
}

User Cancelled Response:

{
  "success": true,
  "operation": "audit",
  "phase": "cancelled_by_user",
  "result": {
    "total_files": 23,
    "issues": {
      "high": 5,
      "medium": 7,
      "low": 3,
      "total": 15
    },
    "discovery_reports_available": true,
    "note": "Spec generation cancelled by user. Discovery reports available for future reference."
  },
  "timestamp": "2025-01-15T12:00:00Z"
}

Discovery Errors:

• Script execution failure: Report which script and error
• No documentation found: Suggest creating docs first
• Permission denied: Report access issue

Spec Generation Errors:

• Spec plugin unavailable: Warn user and explain spec generation requires fractary-spec
• spec-manager invocation fails: Report error and suggest checking spec plugin installation
• Invalid discovery data: Report parsing error
• Cannot write output: Report permission issue

Standards Errors:

• Project standards doc not found: Use plugin defaults only
• Invalid config: Report validation error

Usage Example:

Use the doc-auditor skill to audit documentation:
{
  "operation": "audit",
  "parameters": {
    "project_root": "/path/to/project",
    "output_dir": ".fractary/audit",
    "config_path": ".fractary/plugins/docs/config.json"
  }
}

What to document:

• Final audit report (saved to logs/audits/docs/{timestamp}-documentation-audit.md)
• Discovery results (temporary files in logs/audits/tmp/, cleaned up after report)
• Issues identified by priority
• Standards applied (plugin + project)
• Remediation actions presented for review
• Estimated effort
• User decision (save as spec, refine, or hold off)
• Spec generation results (if user approves)
• Audit log registration via logs-manager

Format:

• Final audit report: Markdown file in logs/audits/docs/{timestamp}-documentation-audit.md
• Audit findings: Formatted text presentation for user review
• Remediation spec: Generated via fractary-spec:spec-manager (if approved)
• Temporary discovery reports: JSON files in logs/audits/tmp/ (cleaned up after report)
• Audit log metadata: Tracked by logs-manager for trend analysis

Log Management:

• All audit runs produce a permanent report in logs/audits/docs/{timestamp}-documentation-audit.md
• Managed by fractary-logs:logs-manager agent
• Temporary discovery files in logs/audits/tmp/ are cleaned up after final report
• Enables tracking of compliance trends over time
• Each audit has a unique timestamp for historical reference