Documentation Testing Skill
Purpose
Provides systematic heuristics for identifying incomplete, outdated, or broken documentation. This skill helps catch the gaps that make onboarding painful for new developers.
Common Documentation Failures
Missing Prerequisites
Projects often assume these are installed without documenting them:
| Category |
Examples |
| JavaScript |
Node.js, npm, yarn, pnpm (and which version) |
| Python |
Python, pip, poetry, conda (and which version) |
| Containers |
Docker, docker-compose, podman |
| Databases |
PostgreSQL, MySQL, Redis, MongoDB, SQLite |
| Cloud CLIs |
aws, gcloud, az, terraform |
| Build tools |
make, cmake, gcc, clang |
| Version managers |
nvm, pyenv, rbenv, asdf |
Missing Environment Configuration
Look for these patterns that indicate undocumented environment variables:
| Language |
Pattern to detect |
| JavaScript/TypeScript |
process.env.XXX |
| Python |
os.environ["XXX"] or os.getenv("XXX") |
| Ruby |
ENV["XXX"] |
| Go |
os.Getenv("XXX") |
| Rust |
std::env::var("XXX") |
Red flags:
- References to
.env files without .env.example
config.yaml with placeholder values but no instructions- Environment variables mentioned in code but not in README
Incomplete Setup Steps
Common missing steps that break new developer onboarding:
| Step |
What's often missing |
| Database setup |
createdb, migrations, seed data |
| Service dependencies |
Starting Redis, Postgres, etc. before the app |
| SSL/TLS |
Local HTTPS setup, certificate generation |
| Host files |
/etc/hosts modifications for local domains |
| Ports |
Required ports and how to check if they're in use |
| Permissions |
File/directory permissions, sudo requirements |
| Git hooks |
Pre-commit hooks that need manual setup |
| IDE setup |
Extensions, settings, debug configurations |
Platform-Specific Gaps
Documentation often assumes one OS. Check for:
| Issue |
Example |
| Shell commands |
sed -i (GNU) vs sed -i '' (BSD/macOS) |
| Path separators |
/ vs \ |
| Package managers |
apt vs brew vs choco vs winget |
| Line endings |
LF vs CRLF issues |
| Case sensitivity |
Filesystem differences |
| Docker |
Docker Desktop vs native Docker |
Validation Sequence
Execute setup validation in this order to catch cascading failures:
1. Prerequisites Check
└─ Are all required tools installed?
└─ Are versions compatible?
2. Repository Setup
└─ Clone works?
└─ Submodules initialized?
└─ Git LFS files pulled?
3. Dependency Installation
└─ npm install / pip install / etc.
└─ Any errors or warnings?
└─ Postinstall scripts succeed?
4. Environment Configuration
└─ .env file created from template?
└─ All required variables filled?
└─ Config files generated?
5. Build Step (if applicable)
└─ Compilation succeeds?
└─ Assets generated?
└─ No TypeScript/type errors?
6. Database/Service Setup
└─ Services started (DB, cache, queue)?
└─ Database created?
└─ Migrations run?
└─ Seed data loaded?
7. Application Start
└─ Dev server starts?
└─ No port conflicts?
└─ No missing config errors?
8. Verification
└─ Health endpoint responds?
└─ Can log in / perform basic action?
└─ Tests pass?
Output Standards
Rate each documentation section with these markers:
| Status |
Meaning |
Action |
| PASS |
Instructions work exactly as written |
None needed |
| AMBIGUOUS |
Instructions work but could confuse newcomers |
Suggest clarification |
| FAIL |
Instructions do not work as written |
Document the bug |
| MISSING |
Expected section does not exist |
Recommend adding |
Documentation Health Report Format
## Documentation Health Report
### Summary
- Total steps tested: N
- Passed: X
- Ambiguous: Y
- Failed: Z
- Missing sections: W
### Detailed Results
#### PASS: [Section Name]
- Step: "Run npm install"
- Result: Completed successfully
#### AMBIGUOUS: [Section Name]
- Step: "Install dependencies"
- Issue: Doesn't specify npm vs yarn
- Suggestion: "Run `npm install` to install dependencies"
#### FAIL: [Section Name]
- Step: "Start the development server"
- Command: `npm run dev`
- Error: `Error: Cannot find module 'dotenv'`
- Documentation Bug: Missing step to create .env file from .env.example
#### MISSING: Database Setup
- Expected: Instructions for database creation and migrations
- Impact: App crashes on start with "relation does not exist"
- Recommendation: Add section with `createdb` and migration commands
Anti-Patterns to Avoid
When testing documentation, do NOT:
- Improvise missing steps - If docs say "install dependencies" without a command, that's a bug
- Use domain knowledge - Pretend you don't know npm from pip
- Skip verification - Always confirm each step actually worked
- Assume success - An empty output might be an error
- Fix the code - Your job is to fix the instructions, not the implementation
1---2name: documentation-testing3description: Documentation Testing Skill4---56# Documentation Testing Skill78## Purpose910Provides systematic heuristics for identifying incomplete, outdated, or broken documentation. This skill helps catch the gaps that make onboarding painful for new developers.1112## Common Documentation Failures1314### Missing Prerequisites1516Projects often assume these are installed without documenting them:1718| Category | Examples |19|----------|----------|20| **JavaScript** | Node.js, npm, yarn, pnpm (and which version) |21| **Python** | Python, pip, poetry, conda (and which version) |22| **Containers** | Docker, docker-compose, podman |23| **Databases** | PostgreSQL, MySQL, Redis, MongoDB, SQLite |24| **Cloud CLIs** | aws, gcloud, az, terraform |25| **Build tools** | make, cmake, gcc, clang |26| **Version managers** | nvm, pyenv, rbenv, asdf |2728### Missing Environment Configuration2930Look for these patterns that indicate undocumented environment variables:3132| Language | Pattern to detect |33|----------|-------------------|34| JavaScript/TypeScript | `process.env.XXX` |35| Python | `os.environ["XXX"]` or `os.getenv("XXX")` |36| Ruby | `ENV["XXX"]` |37| Go | `os.Getenv("XXX")` |38| Rust | `std::env::var("XXX")` |3940**Red flags:**41- References to `.env` files without `.env.example`42- `config.yaml` with placeholder values but no instructions43- Environment variables mentioned in code but not in README4445### Incomplete Setup Steps4647Common missing steps that break new developer onboarding:4849| Step | What's often missing |50|------|---------------------|51| **Database setup** | `createdb`, migrations, seed data |52| **Service dependencies** | Starting Redis, Postgres, etc. before the app |53| **SSL/TLS** | Local HTTPS setup, certificate generation |54| **Host files** | `/etc/hosts` modifications for local domains |55| **Ports** | Required ports and how to check if they're in use |56| **Permissions** | File/directory permissions, sudo requirements |57| **Git hooks** | Pre-commit hooks that need manual setup |58| **IDE setup** | Extensions, settings, debug configurations |5960### Platform-Specific Gaps6162Documentation often assumes one OS. Check for:6364| Issue | Example |65|-------|---------|66| **Shell commands** | `sed -i` (GNU) vs `sed -i ''` (BSD/macOS) |67| **Path separators** | `/` vs `\` |68| **Package managers** | apt vs brew vs choco vs winget |69| **Line endings** | LF vs CRLF issues |70| **Case sensitivity** | Filesystem differences |71| **Docker** | Docker Desktop vs native Docker |7273## Validation Sequence7475Execute setup validation in this order to catch cascading failures:7677```781. Prerequisites Check79 └─ Are all required tools installed?80 └─ Are versions compatible?81822. Repository Setup83 └─ Clone works?84 └─ Submodules initialized?85 └─ Git LFS files pulled?86873. Dependency Installation88 └─ npm install / pip install / etc.89 └─ Any errors or warnings?90 └─ Postinstall scripts succeed?91924. Environment Configuration93 └─ .env file created from template?94 └─ All required variables filled?95 └─ Config files generated?96975. Build Step (if applicable)98 └─ Compilation succeeds?99 └─ Assets generated?100 └─ No TypeScript/type errors?1011026. Database/Service Setup103 └─ Services started (DB, cache, queue)?104 └─ Database created?105 └─ Migrations run?106 └─ Seed data loaded?1071087. Application Start109 └─ Dev server starts?110 └─ No port conflicts?111 └─ No missing config errors?1121138. Verification114 └─ Health endpoint responds?115 └─ Can log in / perform basic action?116 └─ Tests pass?117```118119## Output Standards120121Rate each documentation section with these markers:122123| Status | Meaning | Action |124|--------|---------|--------|125| PASS | Instructions work exactly as written | None needed |126| AMBIGUOUS | Instructions work but could confuse newcomers | Suggest clarification |127| FAIL | Instructions do not work as written | Document the bug |128| MISSING | Expected section does not exist | Recommend adding |129130## Documentation Health Report Format131132```markdown133## Documentation Health Report134135### Summary136- Total steps tested: N137- Passed: X138- Ambiguous: Y139- Failed: Z140- Missing sections: W141142### Detailed Results143144#### PASS: [Section Name]145- Step: "Run npm install"146- Result: Completed successfully147148#### AMBIGUOUS: [Section Name]149- Step: "Install dependencies"150- Issue: Doesn't specify npm vs yarn151- Suggestion: "Run `npm install` to install dependencies"152153#### FAIL: [Section Name]154- Step: "Start the development server"155- Command: `npm run dev`156- Error: `Error: Cannot find module 'dotenv'`157- Documentation Bug: Missing step to create .env file from .env.example158159#### MISSING: Database Setup160- Expected: Instructions for database creation and migrations161- Impact: App crashes on start with "relation does not exist"162- Recommendation: Add section with `createdb` and migration commands163```164165## Anti-Patterns to Avoid166167When testing documentation, do NOT:1681691. **Improvise missing steps** - If docs say "install dependencies" without a command, that's a bug1702. **Use domain knowledge** - Pretend you don't know npm from pip1713. **Skip verification** - Always confirm each step actually worked1724. **Assume success** - An empty output might be an error1735. **Fix the code** - Your job is to fix the instructions, not the implementation
jimmc414