ansible docs-qa-audit
Audit documentation coverage against common user questions. Generates a Q&A matrix, searches docs/code for answers, flags gaps, and creates DRs for missing content.
Trigger
Use when:
- "audit docs", "check documentation coverage", "docs qa"
- "what questions can users answer", "documentation gaps"
- "validate user guides", "check we have answers"
- Preparing for customer demos or releases
Output
Markdown report written by default to docs/reports/docs-qa-audit-YYYY-MM-DD.md.
If invoked with --output path/to/report.md, write the report to that path
instead. The report contains:
- Coverage Matrix — Questions with status (✓ covered / ⚠ partial / ✗ gap)
- Source References — Links to docs/code that answer each question
- Gap Analysis — Missing answers with suggested doc locations
- Action Items — DRs created for gaps, draft answer stubs for documentation
Question Categories
1. What Is It / What Does It Do
- What is APME and what problem does it solve?
- What types of Ansible content can APME scan?
- What rules does APME enforce and why?
- How does APME differ from ansible-lint?
- What are the severity levels and what do they mean?
2. How Can I Use It
- How do I install APME?
- How do I run a basic scan?
- How do I get JSON output for automation?
- How do I run APME in a container?
- How do I use the daemon vs pod mode?
3. Rule Configuration
- How do I enable/disable specific rules?
- How do I change rule severity?
- How do I suppress a rule for a specific line (noqa)?
- How do I configure rules per-project?
- How do I enforce rules and ignore noqa comments?
4. Bring Your Own Rules
- Can I add custom rules?
- How do I write a custom OPA/Rego rule?
- How do I add custom rules via plugins?
- What rule ID conventions should custom rules follow?
5. Demonstrating Value
- How do I show before/after remediation?
- How do I track improvement over time (health score)?
- How do I generate reports for stakeholders?
- How do I measure AI fix acceptance rates?
- What metrics does the Gateway API provide?
6. Integration with Existing Tools
- How do I integrate with GitHub Actions?
- How do I integrate with GitLab CI?
- How do I integrate with Jenkins?
- How do I integrate with Azure DevOps?
- How do I use APME with AAP/AWX?
- How do I use APME with Backstage?
- How do I use APME with pre-commit hooks?
- How do I migrate from ansible-lint?
- How do I use APME in an air-gapped environment?
- How do I integrate with VS Code?
Procedure
See procedure.md for the detailed step-by-step implementation guide.
Overview
- Load Question Set — Use the categories above
- Search Documentation — Glob
docs/**/*.md,README.md,.sdlc/**/*.md - Search Code — Grep for CLI help, config loaders, API endpoints
- Score Coverage:
- ✓ Covered: Clear answer in user-facing docs
- ⚠ Partial: Answer exists but incomplete or only in code/ADRs
- ✗ Gap: No answer found
- Generate Matrix — Markdown table with question, status, source
- Handle Gaps:
- Create DR for each gap (if not already exists)
- Draft answer stub based on code/ADR research
- Suggest target doc file for the answer
- Output Report — Write to
docs/reports/docs-qa-audit-YYYY-MM-DD.md
Search Locations
| Content Type | Locations |
|---|---|
| User guides | docs/guides/*.md |
| README | README.md |
| Rule docs | docs/rules/*.md, src/**/rules/*.md |
| API docs | docs/architecture/*.md, docs/design/*.md, src/apme_gateway/api/ |
| ADRs | .sdlc/adrs/*.md |
| CLI help | src/apme_engine/cli/*.py |
| Config | src/apme_engine/cli/_rules_yml.py, .apme/ patterns |
Example Output
# Documentation Q&A Audit — 2026-05-21
## Coverage Summary
- **Covered**: 18/25 (72%)
- **Partial**: 4/25 (16%)
- **Gaps**: 3/25 (12%)
## Coverage Matrix
| # | Question | Status | Source |
|---|----------|--------|--------|
| 1 | What is APME? | ✓ | [README.md#overview](../../README.md#overview) |
| 2 | How do I install? | ✓ | [README.md#quick-start](../../README.md#quick-start) |
| 3 | How do I write custom OPA rules? | ⚠ | ADR-042 (not user-facing) |
| 4 | How do I use with Backstage? | ✗ | — |
## Gaps & Actions
### Gap: Backstage Integration
- **Question**: How do I use APME with Backstage?
- **DR Created**: DR-018-backstage-integration-guide
- **Suggested Location**: `docs/guides/BACKSTAGE_INTEGRATION.md`
- **Draft Answer**: [See PR #XXX]
Notes
- Run periodically before releases or customer engagements
- Output is designed to feed presentation decks or release-readiness reviews
- Gaps with DRs can be tracked via
/sdlc-status - Link format: When the report lives in
docs/reports/, use../../prefix for repo-root links