Scrape Skill Creator

Scrape → Skill Creator

Turn any website into a reusable Copilot skill by scraping it with agent-browser and authoring a SKILL.md that follows the agent-customization conventions.

When to use

• The user gives a URL (docs, API reference, tutorial, blog post, internal wiki) and wants the agent to package that knowledge as a slash-invocable skill.
• The user asks to "scrape", "crawl", or "ingest" web content into a skill, prompt, or instructions file.
• Existing SKILL.md authoring shortcuts are not enough because the source material lives on the web and must be extracted first.

Do not use this skill for: editing an existing skill (use agent-customization), generic web fetching unrelated to skill creation (use fetch_webpage), or scraping behind auth without explicit user consent.

Prerequisites

Verify agent-browser is installed before scraping:

command -v agent-browser >/dev/null && agent-browser --version || echo "MISSING"

If missing, ask the user before installing. Suggested install (macOS):

brew install agent-browser && agent-browser install
# or: npm install -g agent-browser && agent-browser install

First-time only: agent-browser install downloads Chrome for Testing.

Workflow

1. Clarify inputs

Confirm with the user (use ask-questions tool if available):

• Source URL(s) — single page or a docs root to crawl
• Skill name — kebab-case, ≤64 chars, will become the folder name (default: derive from page title or domain)
• Scope — both (default: write to .agents/skills/<name>/ AND .claude/skills/<name>/ simultaneously), agents-only (.agents/skills/<name>/), claude-only (.claude/skills/<name>/), or a custom comma-separated list of directories
• Crawl scope — see step 2.5; default is "the given page + every sibling page in its sidebar group", not just the given URL
• Auth — does the site need login? If yes, plan to use --profile or --session-name (see agent-browser auth docs)

If the user is vague, infer sensible defaults and proceed; do not block on minor details.

2. Scrape with agent-browser

Fastest path: use the wizard (handles all steps 2–5 automatically):

python3 ./scripts/wizard.py

For manual control, use the batch + JSON flow. The daemon persists between commands.

Single-page extraction:

agent-browser open "<URL>"
agent-browser wait --load networkidle
agent-browser snapshot -i --urls --json > /tmp/skill-snap.json   # interactive elements + link URLs
agent-browser eval "document.body.innerText" --json > /tmp/skill-text.json
agent-browser screenshot /tmp/skill-page.png --full

Preflight matters. Always inject preflight.js before extracting content (auto-done by crawl.py). Without it: collapsed <details>, hidden tab panels, cookie banners, and lazy-loaded lists will all be missing from innerText.

2.5. Discover sidebar / navigation structure (DO NOT SKIP)

Most docs sites (Docusaurus, VitePress, Nextra, GitBook, Mintlify, etc.) put their full information architecture in a left sidebar. The single page you were given is almost always one leaf in a much larger tree. You MUST inspect the sidebar before deciding the skill's scope — otherwise the skill will only know about one API out of dozens.

Detect the sidebar:

# Common sidebar selectors — try them in order until one returns links.
agent-browser eval "
const SELECTORS = [
  'nav[aria-label*=\"sidebar\" i]', 'aside nav', '.sidebar', '.theme-doc-sidebar-container',
  '.VPSidebar', '.nextra-sidebar', '[class*=sidebar i] a', 'aside a', '[role=navigation] a'
];
for (const sel of SELECTORS) {
  const links = [...document.querySelectorAll(sel + (sel.endsWith('a') ? '' : ' a'))]
    .map(a => ({ text: a.innerText.trim(), href: a.href }))
    .filter(l => l.href && !l.href.startsWith('javascript:') && !l.href.includes('#'));
  if (links.length > 3) { console.log(JSON.stringify({ selector: sel, count: links.length, links }, null, 2)); break; }
}
" > /tmp/skill-sidebar.json

If nothing matches, fall back to filtering ALL same-hostname links from snapshot -i --urls --json and clustering by URL prefix (e.g. all links sharing the path prefix of the input URL's parent).

Pick the crawl set based on the sidebar structure:

Always show the user the discovered sidebar tree before crawling >5 pages. Format:

Discovered N pages in sidebar:
  Group A (current): page1, page2, page3
  Group B: page4, page5, ...
  Group C: ...
Proposed crawl: Group A (3 pages). Other groups → suggest as separate skills.
Proceed?

2.6. Crawl the chosen pages — prefer the bundled scripts

This skill ships with a tested pipeline at ./scripts/:

# 1. Crawl every URL listed in urls.json → /tmp/scrape-pages/<slug>__<name>.txt
python3 ./scripts/crawl.py urls.json --out /tmp/scrape-pages

# 2. Build per-group skills with structured references/api.md
python3 ./scripts/gen_skills.py manifest.json \
        --pages /tmp/scrape-pages --out /path/to/skills/

gen_skills.py calls parse_page.py for each module, which decodes the JSON-encoded eval output and structures the page into per-method markdown sections (with 运行环境 tables and fenced ts code blocks). For non-UniAPI sites, edit METHOD_HEAD_RE / SECTION_LABELS / render_env_table in parse_page.py (see scripts/README.md for details).

If you must roll your own loop, follow these rules:

1. Deduplicate URLs, drop the input URL if already scraped, drop fragment-only links (#section), drop obvious assets.
2. Cap at ~20 pages per skill; if more, split into multiple skills (one per sidebar group).
3. For each URL:

   agent-browser open "<url>" && \
     agent-browser wait --load networkidle && \
     agent-browser eval "document.body.innerText" > "/tmp/skill-pages/$(echo "<url>" | shasum | cut -c1-8).txt"
   

4. Use agent-browser batch to chain multiple pages in one CLI call when possible (faster, single daemon round-trip).
5. Close the session when done: agent-browser close --all.

Tips:

• Use agent-browser snapshot -i -c -d 5 to get a compact accessibility tree if innerText is too noisy.
• For SPAs, prefer wait --load networkidle over domcontentloaded.
• For large pages, set AGENT_BROWSER_MAX_OUTPUT=200000 to avoid context overflow.
• Respect robots.txt and the site's terms of service. If the user targets a paywalled or login-gated site, ask before bypassing.

2.7. Check for easily-missed content (DO NOT SKIP)

Before treating a crawl as complete, verify you haven't missed these common omissions:

Run discover.py first to automate most of these checks:

python3 ./scripts/discover.py <seed_url> --out discovered.json
# Then review discovered.json before building urls.json

3. Distill the scraped content

Read the saved files and extract:

• Purpose — what does this site/tool/API do? (1–2 sentences for description)
• Trigger keywords — terms a user might type when they need this knowledge (for skill discovery)
• Core procedures — step-by-step workflows the skill should encode
• Reference data — schemas, command tables, config keys, type lists → save as references/*.md
• Examples — code snippets, request/response samples → save as references/examples.md

Filter aggressively. Strip nav menus, footers, cookie banners, and marketing copy. Keep only what an agent needs to act.

4. Generate the skill

Create the folder structure. By default, output to both .agents/skills/ and .claude/skills/ so the skill is usable by GitHub Copilot agents and Claude Code:

.agents/skills/<skill-name>/       ← for GitHub Copilot / Agents
├── SKILL.md
├── references/
│   └── api.md
.claude/skills/<skill-name>/       ← for Claude Code
├── SKILL.md
├── references/
│   └── api.md

Each directory gets identical content. Structure per skill:

<skill-name>/
├── SKILL.md              # required, name field MUST equal folder name
├── references/
│   ├── <topic-1>.md      # progressive-loaded reference (linked from SKILL.md)
│   └── <topic-2>.md
└── assets/               # optional: templates, sample configs

For SKILL.md, follow the template at skills.md reference — but the canonical rules are:

• Frontmatter: name (matches folder), description (keyword-rich, ≤1024 chars, includes trigger phrases and "Use when…" / "Do not use for…"), optional argument-hint.
• Body: When to use → Workflow / Procedure → References (linked, not inlined).
• Keep SKILL.md under ~500 lines. Push bulk content into references/.
• Use relative paths: [examples](./references/examples.md).

5. Validate

After writing files:

• Confirm folder name === name field in frontmatter.
• Lint frontmatter as YAML (quote any description containing :).
• Verify all ./references/* and ./assets/* links resolve.
• Read the SKILL.md back and check that the description contains discriminating keywords from the source domain.
• Tell the user how to invoke it: /<skill-name> in chat, or describe the trigger phrases.

6. Cleanup

agent-browser close --all   # if no other workflows are using the daemon
rm -f /tmp/skill-snap.json /tmp/skill-text.json /tmp/skill-page.png

Quality checklist

Content coverage

• discover.py was run; special_pages reviewed (types, error codes, changelog, glossary)
• Orphan candidates inspected; relevant ones added to crawl
• Version locked: seed URLs point to a specific version path, not /latest
• Tab panels scraped: preflight.js was injected (done automatically by crawl.py)
• Code examples in references include import statements

Structure

• description contains specific trigger words from the source (not generic "helpful skill")
• description includes both Use when… and Do not use for… clauses
• description has trigger phrases in both languages if docs are non-English
• description lists sibling-skill slugs in Do not use for if multiple skills from the same site
• Reference files are split by topic, each <300 lines
• Reference files have real markdown structure (H3 per method, real tables, real fenced code) — not literal \n characters
• Source URL(s) cited at the top of each reference file

Validation

• doctor.py passes with zero ERRORs
• Folder name === name field in frontmatter
• All relative links in SKILL.md resolve to actual files

Anti-patterns

• Skipping discover.py — you will miss types pages, error code tables, changelogs, orphan best-practice pages, and the version switcher. Always run discovery before committing to a URL list.
• Not locking the version — scraping /latest embeds the current version's content but the URL will serve different content next release. Lock to /v4/... or equivalent.
• Not running preflight — innerText on a page with collapsed tabs/details returns only the visible content. crawl.py injects preflight.js automatically; if you roll your own loop, inject it yourself.
• Missing import statements in examples — doc sites routinely omit the import line. Manually ensure at least one complete example with imports exists per module in references.
• Single-language description — Chinese users won't trigger a skill with only English trigger phrases. Add Chinese key terms.
• Sibling skills without boundary clauses — when you generate N skills from the same site, each description must say what the others cover, so the agent doesn't pick the wrong one.
• Saving raw agent-browser eval stdout to disk — the CLI returns the result JSON-encoded ("...\n..." with literal escape characters) prefixed by ✓ ... status lines. Either pass --json and json.loads(stdout), or run the saved file through ./scripts/parse_page.py which strips the wrapper and decodes for you. Symptom of forgetting: references/*.md full of literal \n and \t.
• Dumping the decoded text without per-method structure — even after decoding, a flat blob of "method name / 入参 / 返回值 / …" is unreadable. Split on the per-method header pattern and emit one ### Module.method section per API; render 运行环境 as a markdown table and 参考示例 as a fenced code block.
• Scraping only the given URL — almost always wrong for docs sites. Always inspect the sidebar (step 2.5) and treat the input URL as a seed, not the scope.
• Dumping innerText into SKILL.md — bloats discovery context; distill first.
• One giant references/all.md — defeats progressive loading; split by topic (one per sidebar entry / API module).
• Skipping the description keywords — the skill won't be auto-invoked.
• Scraping aggressively without confirmation — confirm crawl scope > 5 pages with the user.
• Cramming a whole docs site into one skill — if the sidebar has many groups, propose one skill per group instead of a 5000-line monolith.
• Copying entire copyrighted docs — link to source and summarize procedures instead.