Testing handbook skill generator
Contract
| Field |
Bound contract |
| Trigger |
User wants to discover, generate, refresh, or validate skills from the Trail of Bits Testing Handbook or appsec.guide. |
| Authority |
Reversible local: writes only generated skill directories and their cross-references under the configured output path; rollback is deleting generated directories. No remote mutation. No other state is affected. |
| Side effect |
Local write: generated skill directories (each containing SKILL.md) and cross-reference links between them. No mutation of the handbook source, other plugins, or remote state. |
| Done |
Approved handbook sections produce type-appropriate skills whose references resolve and whose activation and validation checks pass. |
Inputs
- Handbook path (required): local path to the Testing Handbook repository. The
content/docs/ directory must exist. If not supplied, check ./testing-handbook, ../testing-handbook, ~/testing-handbook; then ask the user; then offer to shallow-clone https://github.com/trailofbits/testing-handbook as a last resort.
- Output directory (required): local path where generated skill directories are written. Defaults to the skill's sibling directory.
- Scope (optional): a specific handbook section path to generate from, skipping full discovery. When omitted, the skill runs full discovery across all handbook sections.
Invocation policy is model+human: the model executes discovery, classification, and generation; the human approves the plan before any skill is written.
Refusals
- Will not proceed to generation without explicit user approval of the plan.
- Will not deliver a file over 500 lines: split into sibling files.
- Will not claim the done predicate holds when validation has not passed for a generated skill.
- Will not mutate the handbook source, other plugins, or remote state.
Procedure
- Locate the handbook. Check
./testing-handbook, ../testing-handbook, ~/testing-handbook for a content/docs/ directory. If not found, ask the user for the path. If the user does not know, offer to clone https://github.com/trailofbits/testing-handbook with depth 1. Stop if the handbook cannot be located or content/docs/ is missing. Done when: the handbook is located with a content/docs/ directory.
- Scan directory structure. Walk
{handbook_path}/content/docs/ and enumerate every directory. For each markdown file, parse YAML frontmatter for title, summary, weight, bookCollapseSection, and draft. Done when: every directory and markdown file is enumerated with its frontmatter.
- Classify candidates. For each directory, apply the first matching rule:
/static-analysis/[name]/ with numbered files (00-, 10-): tool skill; /fuzzing/[lang]/[name]/ with index.md or numbered files: fuzzer skill; /fuzzing/techniques/[name]/ with any .md files: technique skill; /crypto/[name]/ with any .md files: domain skill; /web/[name]/ with numbered files or _index.md: tool skill (check exclusions first); _index.md with bookCollapseSection: true: container, scan children, create no skill for the container; any other directory with only _index.md: skip (insufficient content). Most specific (deepest) path wins. When multiple types match, prefer Tool > Fuzzer > Technique > Domain. Done when: every directory is classified or skipped.
- Apply exclusions. Skip a section if:
draft: true appears in frontmatter, the directory is empty, the file is a template or placeholder, or the tool is GUI-only (e.g., web/burp/: Burp Suite requires visual interaction and cannot be operated headlessly). Done when: exclusions are applied and skipped sections are recorded.
- Build candidate list. For each candidate, record: name (slugified from
title), type, source section path, summary from frontmatter, weight, whether a resources file (99-resources.md or 91-resources.md) exists, and related sections. Done when: the candidate list is built with all fields.
- Prioritize candidates. Order by weight field (lower first), then content depth (more numbered files first), then presence of resources file, then core section status (fuzzing, static-analysis first). Done when: the candidate list is ordered.
- Present plan to user. Output a plan table: skill name, source section, type, related sections. List skipped sections with reasons. List external resources to fetch. Wait for explicit user approval before proceeding. Accept modifications: remove skills, change types, rename, add custom related sections. Done when: the user approves the plan.
- Prepare generation context. For each approved skill, collect: primary section content (
_index.md or index.md), numbered files, related sections, and resources. Fetch non-video URLs with a 30-second timeout; skip on timeout and note in warnings. Extract video URLs as title and link only. Verify each candidate's primary content is non-empty, frontmatter has title and summary, and the template for its type exists. Done when: generation context is prepared for every approved skill.
- Pass 1: content generation (parallel). For each approved skill, generate a SKILL.md with all sections except Related Skills. Apply the type-appropriate section structure and Hugo shortcode conversion rules per
references/type-templates.md. Preserve code blocks exactly. Leave a Related Skills placeholder. If content exceeds 450 lines, extract large sections into sibling files and add a decision tree to SKILL.md. Hard limit: 500 lines per file. Done when: every approved skill has a SKILL.md written with its type-appropriate sections.
- Pass 2: cross-reference population (sequential). After all Pass 1 skills are written, list generated skill names. For each skill, determine related skills per the rules in
references/type-templates.md. Replace each placeholder with a Related Skills table. Validate that every referenced skill directory exists. Done when: every placeholder is replaced and all cross-references validate.
- Validate. For each generated skill, verify: YAML frontmatter parses;
name matches ^[a-z0-9-]{1,64}$; description is non-empty and at most 1024 characters; required sections for the skill type are present; line count is under 500; no Hugo shortcodes remain; no escaped backtick sequences remain; all internal links resolve; and all cross-referenced skills exist. Done when: every generated skill passes validation.
- Finalize. Update the repository README with a table of generated skills (author:
testing-handbook-generator). Update the Skills Cross-Reference graph from each skill's Related Skills section. Note any template, discovery, or content extraction issues encountered for future improvement. Done when: the README and cross-reference graph are updated.
Failure and recovery
| Failure class |
Rule |
| Handbook not found |
Stop. Report that no content/docs/ directory exists at the checked locations. Do not proceed to discovery. |
| Clone failure |
Report the error. Ask the user to clone manually and provide the path. |
| Empty or draft section |
Skip the section. Record it in the skipped-sections list with the reason. Continue with remaining candidates. |
| Single agent failure |
Re-run the failed agent alone with the same inputs. Do not re-run the entire parallel batch. |
| Validation failure |
Check the specific failure (missing section, broken reference, shortcode residue). Patch or re-run the single skill. |
| Pass 2 broken reference |
Check whether the referenced skill was skipped. Update or remove the reference. |
| Over 500 lines |
Split into sibling files. Keep SKILL.md as a router with a decision tree. Never deliver a file over 500 lines. |
Partial results are valid: deliver all skills that pass validation and report any that failed or were skipped. Never claim the done predicate holds when validation has not passed for a generated skill.
Rollback: delete the generated skill directories for the current run. No handbook, plugin, or remote state is affected.
Output
Generated skill directories under the configured output path (each containing a SKILL.md with type-appropriate sections, cross-references, and passing validation), an updated README table and cross-reference graph, and a per-skill generation report (line count, split status, populated sections, gaps, warnings, references): ordering: skills, README, graph, reports.
1---2name: testing-handbook-generator3description: Use when the user asks to discover, generate, refresh, or validate skills from the Trail of Bits Testing Handbook or appsec.guide. Not for tasks that require source or remote-system changes.4---56# Testing handbook skill generator78## Contract910| Field | Bound contract |11|---|---|12| Trigger | User wants to discover, generate, refresh, or validate skills from the Trail of Bits Testing Handbook or appsec.guide. |13| Authority | Reversible local: writes only generated skill directories and their cross-references under the configured output path; rollback is deleting generated directories. No remote mutation. No other state is affected. |14| Side effect | Local write: generated skill directories (each containing SKILL.md) and cross-reference links between them. No mutation of the handbook source, other plugins, or remote state. |15| Done | Approved handbook sections produce type-appropriate skills whose references resolve and whose activation and validation checks pass. |1617## Inputs18191. **Handbook path** (required): local path to the Testing Handbook repository. The `content/docs/` directory must exist. If not supplied, check `./testing-handbook`, `../testing-handbook`, `~/testing-handbook`; then ask the user; then offer to shallow-clone `https://github.com/trailofbits/testing-handbook` as a last resort.202. **Output directory** (required): local path where generated skill directories are written. Defaults to the skill's sibling directory.213. **Scope** (optional): a specific handbook section path to generate from, skipping full discovery. When omitted, the skill runs full discovery across all handbook sections.2223Invocation policy is model+human: the model executes discovery, classification, and generation; the human approves the plan before any skill is written.2425## Refusals2627- Will not proceed to generation without explicit user approval of the plan.28- Will not deliver a file over 500 lines: split into sibling files.29- Will not claim the done predicate holds when validation has not passed for a generated skill.30- Will not mutate the handbook source, other plugins, or remote state.3132## Procedure33341. **Locate the handbook.** Check `./testing-handbook`, `../testing-handbook`, `~/testing-handbook` for a `content/docs/` directory. If not found, ask the user for the path. If the user does not know, offer to clone `https://github.com/trailofbits/testing-handbook` with depth 1. Stop if the handbook cannot be located or `content/docs/` is missing. **Done when:** the handbook is located with a `content/docs/` directory.352. **Scan directory structure.** Walk `{handbook_path}/content/docs/` and enumerate every directory. For each markdown file, parse YAML frontmatter for `title`, `summary`, `weight`, `bookCollapseSection`, and `draft`. **Done when:** every directory and markdown file is enumerated with its frontmatter.363. **Classify candidates.** For each directory, apply the first matching rule: `/static-analysis/[name]/` with numbered files (00-, 10-): tool skill; `/fuzzing/[lang]/[name]/` with `index.md` or numbered files: fuzzer skill; `/fuzzing/techniques/[name]/` with any `.md` files: technique skill; `/crypto/[name]/` with any `.md` files: domain skill; `/web/[name]/` with numbered files or `_index.md`: tool skill (check exclusions first); `_index.md` with `bookCollapseSection: true`: container, scan children, create no skill for the container; any other directory with only `_index.md`: skip (insufficient content). Most specific (deepest) path wins. When multiple types match, prefer Tool > Fuzzer > Technique > Domain. **Done when:** every directory is classified or skipped.374. **Apply exclusions.** Skip a section if: `draft: true` appears in frontmatter, the directory is empty, the file is a template or placeholder, or the tool is GUI-only (e.g., `web/burp/`: Burp Suite requires visual interaction and cannot be operated headlessly). **Done when:** exclusions are applied and skipped sections are recorded.385. **Build candidate list.** For each candidate, record: name (slugified from `title`), type, source section path, summary from frontmatter, weight, whether a resources file (`99-resources.md` or `91-resources.md`) exists, and related sections. **Done when:** the candidate list is built with all fields.396. **Prioritize candidates.** Order by weight field (lower first), then content depth (more numbered files first), then presence of resources file, then core section status (fuzzing, static-analysis first). **Done when:** the candidate list is ordered.407. **Present plan to user.** Output a plan table: skill name, source section, type, related sections. List skipped sections with reasons. List external resources to fetch. Wait for explicit user approval before proceeding. Accept modifications: remove skills, change types, rename, add custom related sections. **Done when:** the user approves the plan.418. **Prepare generation context.** For each approved skill, collect: primary section content (`_index.md` or `index.md`), numbered files, related sections, and resources. Fetch non-video URLs with a 30-second timeout; skip on timeout and note in warnings. Extract video URLs as title and link only. Verify each candidate's primary content is non-empty, frontmatter has title and summary, and the template for its type exists. **Done when:** generation context is prepared for every approved skill.429. **Pass 1: content generation (parallel).** For each approved skill, generate a SKILL.md with all sections except Related Skills. Apply the type-appropriate section structure and Hugo shortcode conversion rules per `references/type-templates.md`. Preserve code blocks exactly. Leave a Related Skills placeholder. If content exceeds 450 lines, extract large sections into sibling files and add a decision tree to SKILL.md. Hard limit: 500 lines per file. **Done when:** every approved skill has a SKILL.md written with its type-appropriate sections.4310. **Pass 2: cross-reference population (sequential).** After all Pass 1 skills are written, list generated skill names. For each skill, determine related skills per the rules in `references/type-templates.md`. Replace each placeholder with a Related Skills table. Validate that every referenced skill directory exists. **Done when:** every placeholder is replaced and all cross-references validate.4411. **Validate.** For each generated skill, verify: YAML frontmatter parses; `name` matches `^[a-z0-9-]{1,64}$`; `description` is non-empty and at most 1024 characters; required sections for the skill type are present; line count is under 500; no Hugo shortcodes remain; no escaped backtick sequences remain; all internal links resolve; and all cross-referenced skills exist. **Done when:** every generated skill passes validation.4512. **Finalize.** Update the repository README with a table of generated skills (author: `testing-handbook-generator`). Update the Skills Cross-Reference graph from each skill's Related Skills section. Note any template, discovery, or content extraction issues encountered for future improvement. **Done when:** the README and cross-reference graph are updated.4647## Failure and recovery4849| Failure class | Rule |50|---|---|51| Handbook not found | Stop. Report that no `content/docs/` directory exists at the checked locations. Do not proceed to discovery. |52| Clone failure | Report the error. Ask the user to clone manually and provide the path. |53| Empty or draft section | Skip the section. Record it in the skipped-sections list with the reason. Continue with remaining candidates. |54| Single agent failure | Re-run the failed agent alone with the same inputs. Do not re-run the entire parallel batch. |55| Validation failure | Check the specific failure (missing section, broken reference, shortcode residue). Patch or re-run the single skill. |56| Pass 2 broken reference | Check whether the referenced skill was skipped. Update or remove the reference. |57| Over 500 lines | Split into sibling files. Keep SKILL.md as a router with a decision tree. Never deliver a file over 500 lines. |5859Partial results are valid: deliver all skills that pass validation and report any that failed or were skipped. Never claim the done predicate holds when validation has not passed for a generated skill.6061Rollback: delete the generated skill directories for the current run. No handbook, plugin, or remote state is affected.6263## Output6465Generated skill directories under the configured output path (each containing a SKILL.md with type-appropriate sections, cross-references, and passing validation), an updated README table and cross-reference graph, and a per-skill generation report (line count, split status, populated sections, gaps, warnings, references): ordering: skills, README, graph, reports.
OutlineDriven