QA Spec Auditor
Purpose
Compare "how it should be" vs "how it is" after each release or sprint. Detect spec drift, outdated documentation, undocumented features, and regression set optimization opportunities. Produce actionable drift reports with task suggestions for qa-task-creator.
Trigger Phrases
- "Spec audit" / "Documentation audit for [release/sprint]"
- "Spec drift" / "Detect drift between docs and implementation"
- "Outdated documentation" / "Docs vs implementation comparison"
- "Undocumented features" / "Find features not in specs"
- "Requirements vs UI" / "Requirements vs API contract"
- "Living documentation audit" / "Spec compliance check"
- "Regression set optimization" / "Worn out tests" / "Test rotation"
Comparison Dimensions
| Dimension |
Source A |
Source B |
Detects |
| Requirements ↔ UI/behavior |
Requirements, specs |
Live UI (Playwright MCP) |
UI doesn't match spec; missing flows |
| Requirements ↔ API contract |
Requirements, specs |
OpenAPI, actual endpoints |
API drift; undocumented endpoints |
| Requirements ↔ Test results |
Requirements |
Test cases, RTM, execution |
Untested requirements; orphan tests |
| Documentation ↔ Implementation |
Docs, README, wikis |
Code, UI, API |
Outdated docs; undocumented behavior |
See references/audit-checklist.md for checklists per dimension.
Detection Categories
| Category |
Description |
Output Action |
| Outdated documentation |
Docs describe old behavior |
Update docs |
| Undocumented features |
Implementation has features not in specs |
Add to specs or create tests |
| Spec drift |
Behavior changed but docs not updated |
Update docs or fix implementation |
| Regression set optimization |
"Worn out" tests (always pass, low value); suggest rotation |
Suggest test rotation; prioritize high-value tests |
See references/drift-patterns.md for common patterns and remediation.
Output Deliverables
- Drift Report — Per dimension: findings, severity, affected artifacts
- Action Items — Update docs / fix implementation / create tests
- Task Suggestions — Structured for qa-task-creator (doc updates, test creation, spec fixes)
- Regression Set Recommendations — Tests to rotate, prioritize, or deprioritize
Drift Report Template
# Spec Audit — [Release/Sprint]
## Summary
| Dimension | Findings | Critical | Action Items |
|-----------|----------|----------|--------------|
| Req ↔ UI | N | X | [Count] |
| Req ↔ API | N | X | [Count] |
| Req ↔ Tests | N | X | [Count] |
| Docs ↔ Impl | N | X | [Count] |
## Findings by Dimension
### Requirements ↔ UI
- [Finding] — [Severity] — [Action]
### Requirements ↔ API
- [Finding] — [Severity] — [Action]
### Requirements ↔ Tests
- [Finding] — [Severity] — [Action]
### Documentation ↔ Implementation
- [Finding] — [Severity] — [Action]
## Task Suggestions (for qa-task-creator)
1. [Task type] — [Description] — [Owner]
2. ...
## Regression Set Recommendations
- Rotate: [Tests that are "worn out"]
- Prioritize: [High-value tests]
- Deprioritize: [Low-value, stable tests]
Workflow
- Input: Requirements, specs, OpenAPI/API docs, test cases/RTM, optional UI URL
- Requirements ↔ UI: Use Playwright MCP to explore UI; compare flows, elements, validation to spec
- Requirements ↔ API: Compare OpenAPI/spec to requirements; detect undocumented endpoints, schema drift
- Requirements ↔ Tests: Validate RTM; find untested requirements, orphan tests
- Documentation ↔ Implementation: Compare docs to code/UI/API; flag outdated sections
- Regression optimization: Analyze test history; identify always-pass, low-value tests; suggest rotation
- Output: Drift report, action items, task suggestions for qa-task-creator
MCP Tools Used
- Playwright MCP (cursor-ide-browser): UI exploration, compare live behavior to spec
- GitHub MCP: Change tracking, PR/commit history for docs and code
Integration with Other Skills
| Need |
Skill |
Usage |
| Task creation |
qa-task-creator |
Convert action items to structured tasks |
| Requirements source |
qa-requirements-generator |
Normalized requirements for comparison |
| API contract |
qa-api-contract-curator |
OpenAPI for API dimension |
| Test cases/RTM |
qa-testcase-from-docs, qa-coverage-analyzer |
Test mapping, coverage |
| Spec writing |
qa-spec-writer |
Update specs when drift found |
Scope
Can do (autonomous):
- Compare requirements vs UI (via Playwright MCP)
- Compare requirements vs API contract (OpenAPI)
- Compare requirements vs test results (RTM, coverage)
- Compare documentation vs implementation
- Detect outdated docs, undocumented features, spec drift
- Produce drift report with action items
- Suggest tasks for qa-task-creator
- Recommend regression set optimization (test rotation)
- Use GitHub MCP for change tracking
Cannot do (requires confirmation):
- Modify documentation or code directly
- Change requirements without stakeholder approval
- Override organizational doc/spec policies
Will not do (out of scope):
- Execute tests (consume results only)
- Write or modify test automation
- Deploy or change production systems
Quality Checklist
Troubleshooting
| Symptom |
Likely Cause |
Fix |
| UI comparison fails |
Playwright MCP not available; auth required |
Use API/docs comparison only; note "UI audit skipped" |
| API contract missing |
No OpenAPI; endpoints undocumented |
Use qa-api-contract-curator first; manual endpoint list |
| RTM incomplete |
Requirements lack IDs; tests not traced |
Use qa-requirements-generator; qa-testcase-from-docs for traceability |
| Too many findings |
Broad scope; noisy comparison |
Focus on critical modules; prioritize by business impact |
| Regression data missing |
No test history |
Use execution logs; flag "insufficient data" |
| GitHub MCP errors |
Token, permissions |
Verify GitHub token; check repo access |
Reference Files
| Topic |
Reference |
| Audit checklist per dimension |
references/audit-checklist.md |
| Drift patterns and remediation |
references/drift-patterns.md |
1---2name: qa-spec-auditor3description: Living documentation auditor that compares requirements vs UI/behavior vs API contract vs test results to detect spec drift, outdated docs, and undocumented features.4---56# QA Spec Auditor78## Purpose910Compare "how it should be" vs "how it is" after each release or sprint. Detect spec drift, outdated documentation, undocumented features, and regression set optimization opportunities. Produce actionable drift reports with task suggestions for qa-task-creator.1112## Trigger Phrases1314- "Spec audit" / "Documentation audit for [release/sprint]"15- "Spec drift" / "Detect drift between docs and implementation"16- "Outdated documentation" / "Docs vs implementation comparison"17- "Undocumented features" / "Find features not in specs"18- "Requirements vs UI" / "Requirements vs API contract"19- "Living documentation audit" / "Spec compliance check"20- "Regression set optimization" / "Worn out tests" / "Test rotation"2122## Comparison Dimensions2324| Dimension | Source A | Source B | Detects |25|-----------|----------|----------|---------|26| **Requirements ↔ UI/behavior** | Requirements, specs | Live UI (Playwright MCP) | UI doesn't match spec; missing flows |27| **Requirements ↔ API contract** | Requirements, specs | OpenAPI, actual endpoints | API drift; undocumented endpoints |28| **Requirements ↔ Test results** | Requirements | Test cases, RTM, execution | Untested requirements; orphan tests |29| **Documentation ↔ Implementation** | Docs, README, wikis | Code, UI, API | Outdated docs; undocumented behavior |3031See `references/audit-checklist.md` for checklists per dimension.3233## Detection Categories3435| Category | Description | Output Action |36|----------|-------------|---------------|37| **Outdated documentation** | Docs describe old behavior | Update docs |38| **Undocumented features** | Implementation has features not in specs | Add to specs or create tests |39| **Spec drift** | Behavior changed but docs not updated | Update docs or fix implementation |40| **Regression set optimization** | "Worn out" tests (always pass, low value); suggest rotation | Suggest test rotation; prioritize high-value tests |4142See `references/drift-patterns.md` for common patterns and remediation.4344## Output Deliverables45461. **Drift Report** — Per dimension: findings, severity, affected artifacts472. **Action Items** — Update docs / fix implementation / create tests483. **Task Suggestions** — Structured for qa-task-creator (doc updates, test creation, spec fixes)494. **Regression Set Recommendations** — Tests to rotate, prioritize, or deprioritize5051### Drift Report Template5253```markdown54# Spec Audit — [Release/Sprint]5556## Summary57| Dimension | Findings | Critical | Action Items |58|-----------|----------|----------|--------------|59| Req ↔ UI | N | X | [Count] |60| Req ↔ API | N | X | [Count] |61| Req ↔ Tests | N | X | [Count] |62| Docs ↔ Impl | N | X | [Count] |6364## Findings by Dimension65### Requirements ↔ UI66- [Finding] — [Severity] — [Action]67### Requirements ↔ API68- [Finding] — [Severity] — [Action]69### Requirements ↔ Tests70- [Finding] — [Severity] — [Action]71### Documentation ↔ Implementation72- [Finding] — [Severity] — [Action]7374## Task Suggestions (for qa-task-creator)751. [Task type] — [Description] — [Owner]762. ...7778## Regression Set Recommendations79- Rotate: [Tests that are "worn out"]80- Prioritize: [High-value tests]81- Deprioritize: [Low-value, stable tests]82```8384## Workflow85861. **Input:** Requirements, specs, OpenAPI/API docs, test cases/RTM, optional UI URL872. **Requirements ↔ UI:** Use Playwright MCP to explore UI; compare flows, elements, validation to spec883. **Requirements ↔ API:** Compare OpenAPI/spec to requirements; detect undocumented endpoints, schema drift894. **Requirements ↔ Tests:** Validate RTM; find untested requirements, orphan tests905. **Documentation ↔ Implementation:** Compare docs to code/UI/API; flag outdated sections916. **Regression optimization:** Analyze test history; identify always-pass, low-value tests; suggest rotation927. **Output:** Drift report, action items, task suggestions for qa-task-creator9394## MCP Tools Used9596- **Playwright MCP (cursor-ide-browser):** UI exploration, compare live behavior to spec97- **GitHub MCP:** Change tracking, PR/commit history for docs and code9899## Integration with Other Skills100101| Need | Skill | Usage |102|------|-------|-------|103| Task creation | qa-task-creator | Convert action items to structured tasks |104| Requirements source | qa-requirements-generator | Normalized requirements for comparison |105| API contract | qa-api-contract-curator | OpenAPI for API dimension |106| Test cases/RTM | qa-testcase-from-docs, qa-coverage-analyzer | Test mapping, coverage |107| Spec writing | qa-spec-writer | Update specs when drift found |108109## Scope110111**Can do (autonomous):**112- Compare requirements vs UI (via Playwright MCP)113- Compare requirements vs API contract (OpenAPI)114- Compare requirements vs test results (RTM, coverage)115- Compare documentation vs implementation116- Detect outdated docs, undocumented features, spec drift117- Produce drift report with action items118- Suggest tasks for qa-task-creator119- Recommend regression set optimization (test rotation)120- Use GitHub MCP for change tracking121122**Cannot do (requires confirmation):**123- Modify documentation or code directly124- Change requirements without stakeholder approval125- Override organizational doc/spec policies126127**Will not do (out of scope):**128- Execute tests (consume results only)129- Write or modify test automation130- Deploy or change production systems131132## Quality Checklist133134- [ ] All four comparison dimensions addressed135- [ ] Drift report includes severity per finding136- [ ] Action items are specific (update X, fix Y, create test Z)137- [ ] Task suggestions formatted for qa-task-creator138- [ ] Regression recommendations include rationale139- [ ] No hardcoded secrets; URLs/config referenced140141## Troubleshooting142143| Symptom | Likely Cause | Fix |144|---------|--------------|-----|145| UI comparison fails | Playwright MCP not available; auth required | Use API/docs comparison only; note "UI audit skipped" |146| API contract missing | No OpenAPI; endpoints undocumented | Use qa-api-contract-curator first; manual endpoint list |147| RTM incomplete | Requirements lack IDs; tests not traced | Use qa-requirements-generator; qa-testcase-from-docs for traceability |148| Too many findings | Broad scope; noisy comparison | Focus on critical modules; prioritize by business impact |149| Regression data missing | No test history | Use execution logs; flag "insufficient data" |150| GitHub MCP errors | Token, permissions | Verify GitHub token; check repo access |151152## Reference Files153154| Topic | Reference |155|-------|-----------|156| Audit checklist per dimension | `references/audit-checklist.md` |157| Drift patterns and remediation | `references/drift-patterns.md` |
azanir