# Scrape Skill Creator

> Scrape a website's knowledge with agent-browser and turn it into a new SKILL.md. Use when the user wants to: create a skill from a documentation site, blog, API reference, tutorial, or any URL; bootstrap a SKILL.md from web content; package external knowledge as a Copilot skill. Trigger phrases: '从这个网站做一个 skill', 'scrape this site into a skill', '把这个文档变成 skill', 'create a skill from <url>', 'turn <url> into a skill', 'agent-browser skill creator'. Combines web scraping (agent-browser CLI) with skill authoring (agent-customization conventions).

- Skill: `nealst/scrape-skill-creator` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add nealst/scrape-skill-creator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nealst/scrape-skill-creator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: nealst (https://skillmd.com/u/nealst)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nealst/scrape-skill-creator

---


# Scrape → Skill Creator

Turn any website into a reusable Copilot skill by scraping it with [agent-browser](https://github.com/vercel-labs/agent-browser) and authoring a SKILL.md that follows the `agent-customization` conventions.

## When to use

- The user gives a URL (docs, API reference, tutorial, blog post, internal wiki) and wants the agent to package that knowledge as a slash-invocable skill.
- The user asks to "scrape", "crawl", or "ingest" web content into a skill, prompt, or instructions file.
- Existing SKILL.md authoring shortcuts are not enough because the source material lives on the web and must be extracted first.

Do **not** use this skill for: editing an existing skill (use `agent-customization`), generic web fetching unrelated to skill creation (use `fetch_webpage`), or scraping behind auth without explicit user consent.

## Prerequisites

Verify `agent-browser` is installed before scraping:

```bash
command -v agent-browser >/dev/null && agent-browser --version || echo "MISSING"
```

If missing, ask the user before installing. Suggested install (macOS):

```bash
brew install agent-browser && agent-browser install
# or: npm install -g agent-browser && agent-browser install
```

First-time only: `agent-browser install` downloads Chrome for Testing.

## Workflow

### 1. Clarify inputs

Confirm with the user (use ask-questions tool if available):

- **Source URL(s)** — single page or a docs root to crawl
- **Skill name** — kebab-case, ≤64 chars, will become the folder name (default: derive from page title or domain)
- **Scope** — `both` (default: write to `.agents/skills/<name>/` AND `.claude/skills/<name>/` simultaneously), `agents-only` (`.agents/skills/<name>/`), `claude-only` (`.claude/skills/<name>/`), or a custom comma-separated list of directories
- **Crawl scope** — see step 2.5; default is **"the given page + every sibling page in its sidebar group"**, not just the given URL
- **Auth** — does the site need login? If yes, plan to use `--profile` or `--session-name` (see [agent-browser auth docs](https://github.com/vercel-labs/agent-browser#authentication))

If the user is vague, infer sensible defaults and proceed; do not block on minor details.

### 2. Scrape with agent-browser

**Fastest path: use the wizard** (handles all steps 2–5 automatically):

```bash
python3 ./scripts/wizard.py
```

For manual control, use the **batch + JSON** flow. The daemon persists between commands.

**Single-page extraction:**

```bash
agent-browser open "<URL>"
agent-browser wait --load networkidle
agent-browser snapshot -i --urls --json > /tmp/skill-snap.json   # interactive elements + link URLs
agent-browser eval "document.body.innerText" --json > /tmp/skill-text.json
agent-browser screenshot /tmp/skill-page.png --full
```

> **Preflight matters.** Always inject `preflight.js` before extracting content (auto-done by `crawl.py`). Without it: collapsed `<details>`, hidden tab panels, cookie banners, and lazy-loaded lists will all be missing from `innerText`.

### 2.5. Discover sidebar / navigation structure (DO NOT SKIP)

Most docs sites (Docusaurus, VitePress, Nextra, GitBook, Mintlify, etc.) put their full information architecture in a **left sidebar**. The single page you were given is almost always one leaf in a much larger tree. **You MUST inspect the sidebar before deciding the skill's scope** — otherwise the skill will only know about one API out of dozens.

**Detect the sidebar:**

```bash
# Common sidebar selectors — try them in order until one returns links.
agent-browser eval "
const SELECTORS = [
  'nav[aria-label*=\"sidebar\" i]', 'aside nav', '.sidebar', '.theme-doc-sidebar-container',
  '.VPSidebar', '.nextra-sidebar', '[class*=sidebar i] a', 'aside a', '[role=navigation] a'
];
for (const sel of SELECTORS) {
  const links = [...document.querySelectorAll(sel + (sel.endsWith('a') ? '' : ' a'))]
    .map(a => ({ text: a.innerText.trim(), href: a.href }))
    .filter(l => l.href && !l.href.startsWith('javascript:') && !l.href.includes('#'));
  if (links.length > 3) { console.log(JSON.stringify({ selector: sel, count: links.length, links }, null, 2)); break; }
}
" > /tmp/skill-sidebar.json
```

If nothing matches, fall back to filtering ALL same-hostname links from `snapshot -i --urls --json` and clustering by URL prefix (e.g. all links sharing the path prefix of the input URL's parent).

**Pick the crawl set** based on the sidebar structure:

| Sidebar shape | Recommended scope |
|---|---|
| Flat list (≤30 links) | Crawl all of them — one skill covers the whole product |
| Grouped (input URL is in a group of N siblings) | Crawl the **current group** (e.g. all "Router/Navigator/…" siblings); other groups become candidates for separate skills |
| Huge tree (>50 links across many groups) | Crawl the input URL's group only; **report the other groups to the user** and ask if they want one skill per group |

**Always show the user the discovered sidebar tree before crawling >5 pages.** Format:

```
Discovered N pages in sidebar:
  Group A (current): page1, page2, page3
  Group B: page4, page5, ...
  Group C: ...
Proposed crawl: Group A (3 pages). Other groups → suggest as separate skills.
Proceed?
```

### 2.6. Crawl the chosen pages — prefer the bundled scripts

This skill ships with a tested pipeline at [./scripts/](./scripts/README.md):

```bash
# 1. Crawl every URL listed in urls.json → /tmp/scrape-pages/<slug>__<name>.txt
python3 ./scripts/crawl.py urls.json --out /tmp/scrape-pages

# 2. Build per-group skills with structured references/api.md
python3 ./scripts/gen_skills.py manifest.json \
        --pages /tmp/scrape-pages --out /path/to/skills/
```

`gen_skills.py` calls `parse_page.py` for each module, which **decodes the
JSON-encoded eval output** and **structures the page into per-method
markdown sections** (with `运行环境` tables and fenced `ts` code blocks).
For non-UniAPI sites, edit `METHOD_HEAD_RE` / `SECTION_LABELS` /
`render_env_table` in `parse_page.py` (see scripts/README.md for details).

If you must roll your own loop, follow these rules:

1. Deduplicate URLs, drop the input URL if already scraped, drop fragment-only links (`#section`), drop obvious assets.
2. Cap at ~20 pages per skill; if more, split into multiple skills (one per sidebar group).
3. For each URL:
   ```bash
   agent-browser open "<url>" && \
     agent-browser wait --load networkidle && \
     agent-browser eval "document.body.innerText" > "/tmp/skill-pages/$(echo "<url>" | shasum | cut -c1-8).txt"
   ```
4. Use `agent-browser batch` to chain multiple pages in one CLI call when possible (faster, single daemon round-trip).
5. Close the session when done: `agent-browser close --all`.

**Tips:**
- Use `agent-browser snapshot -i -c -d 5` to get a compact accessibility tree if `innerText` is too noisy.
- For SPAs, prefer `wait --load networkidle` over `domcontentloaded`.
- For large pages, set `AGENT_BROWSER_MAX_OUTPUT=200000` to avoid context overflow.
- Respect `robots.txt` and the site's terms of service. If the user targets a paywalled or login-gated site, ask before bypassing.

### 2.7. Check for easily-missed content (DO NOT SKIP)

Before treating a crawl as complete, verify you haven't missed these common omissions:

| Often-missed content | Why it matters | How to find it |
|---|---|---|
| **Version switcher** | Same URL serves different content per version; half a year later the skill is wrong | `discover.py` detects `[class*=version]` selectors and reports active version |
| **Tab panels / collapsibles** | `innerText` only returns the *active* tab; other panels are invisible | `preflight.js` clicks all tabs before extraction (auto-injected by `crawl.py`) |
| **Lazy-loaded content** | Only first-screen items scraped | `preflight.js` scrolls to bottom (auto) |
| **Types / Interfaces / Enums page** | Method signatures become black boxes without type definitions | `discover.py` → `special_pages.types` |
| **Error code table** | `catch` blocks are empty without error names/meanings | `discover.py` → `special_pages.error_codes` |
| **Changelog / Migration guide** | Agent picks deprecated API without this | `discover.py` → `special_pages.changelog` |
| **Orphan pages** | Frequent internal links not in sidebar (e.g., "Best practices", "Glossary") | `discover.py` → `orphan_candidates` (freq ≥ 3) |
| **Footer links** | Internal links that docs teams add to footer but not sidebar | `discover.py` → `footer_links` |
| **Code examples without `import`** | Docs omit imports; agent generates broken code | Manually ensure references include import lines |
| **Multi-language triggers** | Chinese doc + English-only `description` → skill never fires on Chinese queries | Add both-language trigger phrases in `description` |
| **Cross-skill boundary** | Multiple sibling skills, agent mixes them up | Each `description` must include `Do not use for: <sibling-skill-slugs>` |

**Run `discover.py` first to automate most of these checks:**

```bash
python3 ./scripts/discover.py <seed_url> --out discovered.json
# Then review discovered.json before building urls.json
```

### 3. Distill the scraped content

Read the saved files and extract:

- **Purpose** — what does this site/tool/API do? (1–2 sentences for `description`)
- **Trigger keywords** — terms a user might type when they need this knowledge (for skill discovery)
- **Core procedures** — step-by-step workflows the skill should encode
- **Reference data** — schemas, command tables, config keys, type lists → save as `references/*.md`
- **Examples** — code snippets, request/response samples → save as `references/examples.md`

**Filter aggressively.** Strip nav menus, footers, cookie banners, and marketing copy. Keep only what an agent needs to *act*.

### 4. Generate the skill

Create the folder structure. By default, output to **both** `.agents/skills/` and `.claude/skills/` so the skill is usable by GitHub Copilot agents and Claude Code:

```
.agents/skills/<skill-name>/       ← for GitHub Copilot / Agents
├── SKILL.md
├── references/
│   └── api.md
.claude/skills/<skill-name>/       ← for Claude Code
├── SKILL.md
├── references/
│   └── api.md
```

Each directory gets identical content. Structure per skill:

```
<skill-name>/
├── SKILL.md              # required, name field MUST equal folder name
├── references/
│   ├── <topic-1>.md      # progressive-loaded reference (linked from SKILL.md)
│   └── <topic-2>.md
└── assets/               # optional: templates, sample configs
```

For `SKILL.md`, follow the template at [skills.md reference](../../.vscode/extensions/github.copilot-chat-*/assets/prompts/skills/agent-customization/references/skills.md) — but the canonical rules are:

- Frontmatter: `name` (matches folder), `description` (keyword-rich, ≤1024 chars, includes trigger phrases and "Use when…" / "Do not use for…"), optional `argument-hint`.
- Body: **When to use** → **Workflow / Procedure** → **References** (linked, not inlined).
- Keep `SKILL.md` under ~500 lines. Push bulk content into `references/`.
- Use relative paths: `[examples](./references/examples.md)`.

### 5. Validate

After writing files:

- Confirm folder name === `name` field in frontmatter.
- Lint frontmatter as YAML (quote any `description` containing `:`).
- Verify all `./references/*` and `./assets/*` links resolve.
- Read the SKILL.md back and check that the description contains discriminating keywords from the source domain.
- Tell the user how to invoke it: `/<skill-name>` in chat, or describe the trigger phrases.

### 6. Cleanup

```bash
agent-browser close --all   # if no other workflows are using the daemon
rm -f /tmp/skill-snap.json /tmp/skill-text.json /tmp/skill-page.png
```

## Quality checklist

**Content coverage**
- [ ] `discover.py` was run; `special_pages` reviewed (types, error codes, changelog, glossary)
- [ ] Orphan candidates inspected; relevant ones added to crawl
- [ ] Version locked: seed URLs point to a specific version path, not `/latest`
- [ ] Tab panels scraped: `preflight.js` was injected (done automatically by `crawl.py`)
- [ ] Code examples in references include `import` statements

**Structure**
- [ ] `description` contains specific trigger words from the source (not generic "helpful skill")
- [ ] `description` includes both **Use when…** and **Do not use for…** clauses
- [ ] `description` has trigger phrases in both languages if docs are non-English
- [ ] `description` lists sibling-skill slugs in **Do not use for** if multiple skills from the same site
- [ ] Reference files are split by topic, each <300 lines
- [ ] Reference files have **real markdown structure** (H3 per method, real tables, real fenced code) — not literal `\n` characters
- [ ] Source URL(s) cited at the top of each reference file

**Validation**
- [ ] `doctor.py` passes with zero ERRORs
- [ ] Folder name === `name` field in frontmatter
- [ ] All relative links in SKILL.md resolve to actual files

## Anti-patterns

- **Skipping `discover.py`** — you will miss types pages, error code tables, changelogs, orphan best-practice pages, and the version switcher. Always run discovery before committing to a URL list.
- **Not locking the version** — scraping `/latest` embeds the current version's content but the URL will serve different content next release. Lock to `/v4/...` or equivalent.
- **Not running preflight** — `innerText` on a page with collapsed tabs/details returns only the visible content. `crawl.py` injects `preflight.js` automatically; if you roll your own loop, inject it yourself.
- **Missing `import` statements in examples** — doc sites routinely omit the import line. Manually ensure at least one complete example with imports exists per module in references.
- **Single-language `description`** — Chinese users won't trigger a skill with only English trigger phrases. Add Chinese key terms.
- **Sibling skills without boundary clauses** — when you generate N skills from the same site, each `description` must say what the *others* cover, so the agent doesn't pick the wrong one.
- **Saving raw `agent-browser eval` stdout to disk** — the CLI returns the result **JSON-encoded** (`"...\n..."` with literal escape characters) prefixed by `✓ ...` status lines. Either pass `--json` and `json.loads(stdout)`, or run the saved file through [./scripts/parse_page.py](./scripts/parse_page.py) which strips the wrapper and decodes for you. Symptom of forgetting: `references/*.md` full of literal `\n` and `\t`.
- **Dumping the decoded text without per-method structure** — even after decoding, a flat blob of "method name / 入参 / 返回值 / …" is unreadable. Split on the per-method header pattern and emit one `### Module.method` section per API; render `运行环境` as a markdown table and `参考示例` as a fenced code block.
- **Scraping only the given URL** — almost always wrong for docs sites. Always inspect the sidebar (step 2.5) and treat the input URL as a *seed*, not the *scope*.
- **Dumping `innerText` into SKILL.md** — bloats discovery context; distill first.
- **One giant references/all.md** — defeats progressive loading; split by topic (one per sidebar entry / API module).
- **Skipping the `description` keywords** — the skill won't be auto-invoked.
- **Scraping aggressively without confirmation** — confirm crawl scope > 5 pages with the user.
- **Cramming a whole docs site into one skill** — if the sidebar has many groups, propose one skill per group instead of a 5000-line monolith.
- **Copying entire copyrighted docs** — link to source and summarize procedures instead.