How do I install the Web Crawl skill?

Run npx skillmds@latest add anubisrooster/web-crawl in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Web Crawl safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Web Crawl?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Web Crawl free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Web Crawl?

AnubisRooster (@anubisrooster) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Web Crawl by anubisrooster — Agent Skill

Name: Web Crawl
Author: AnubisRooster

Web Crawl

Crawl an external website to discover and log bugs. Fetches pages, extracts links, follows navigation up to a configurable depth, and flags pages whose titles/URLs contain bug-related keywords. Outputs a CSV of discovered pages with bug-indicator, title, and source URL.

AnubisRooster Updated Sep 21, 2026

File contents

/web-crawl

Crawl an external website to discover and log bugs.

Usage

/web-crawl <url>                           # crawl start URL (default depth 3, max 50 pages)
/web-crawl <url> --max-pages N           # stop after N pages
/web-crawl <url> --max-depth N           # stop following links at depth N
/web-crawl <url> --bug-keywords "kw1,kw2" # only flag pages with these words in title/URL
/web-crawl <url> --no-recurse            # only fetch the start URL, no link following

What it does

Fetches the start URL (and optional follow-up pages) using the web fetch tool
Extracts all hyperlinks (<a href>) from each page
Follows links up to --max-depth or until --max-pages is reached
Flags pages whose <title> or URL contains any of the --bug-keywords (default: bug,issue,fix,todo,changelog)
Writes a CSV file web-crawl-results.csv in the current directory with columns:
- depth — link hop number (0 = start URL)
- url — the page URL
- title — <title> tag content, or fallback Untitled
- bug-indicator — YES if any bug keyword matched, NO otherwise
- source — the URL where the link was discovered

How to use

Invoke /web-crawl https://example.com --max-pages 20 --bug-keywords "bug,issue,task"
The skill writes web-crawl-results.csv to the current working directory
Open the CSV to review discovered pages; rows with bug-indicator=YES are likely bug-related
Re-run with different keywords or a deeper max-depth to broaden/narrow the scan

Tips

External sites may block aggressive crawling; keep --max-pages moderate (20–50) and --max-depth low (2–3)
If a page requires JavaScript rendering, this skill cannot see it — it only reads static HTML
Use --no-recurse to just validate the start URL's title/links without following anything
The CSV can be imported into spreadsheets, databases, or issue trackers

Caveats

Only follows <a href> links; does not render CSS/JS or interact with forms
Respects robots.txt implicitly (no guarantee) — keep crawling responsible
Same-domain only: links pointing outside the start URL's host are ignored to avoid infinite cross-site crawling
If --bug-keywords match only the URL path (not the title), consider using more specific terms that appear in bug trackers

Crawling JavaScript SPAs

Static fetches return an empty shell for React/Vue/Angular sites (no links, no content - just a <div id="root">). For those, use the companion Playwright renderer crawl_spa.py in this folder:

python crawl_spa.py https://corp.example.com ./example-crawl 30

Requires pip install playwright && playwright install chromium
Renders each route headlessly, extracts main-content text, follows same-host links BFS, and saves one markdown file per page (with YAML frontmatter: title, url, captured_at) plus a manifest.json - ready for graphify ingestion.
Choose by site type: web-crawl.ps1 for server-rendered sites, crawl_spa.py when the first fetch returns an empty shell.

---name: web-crawldescription: Crawl an external website to discover and log bugs. Fetches pages, extracts links, follows navigation up to a configurable depth, and flags pages whose titles/URLs contain bug-related keywords. Outputs a CSV of discovered pages with bug-indicator, title, and source URL.---# /web-crawlCrawl an external website to discover and log bugs.## Usage```/web-crawl <url> # crawl start URL (default depth 3, max 50 pages)/web-crawl <url> --max-pages N # stop after N pages/web-crawl <url> --max-depth N # stop following links at depth N/web-crawl <url> --bug-keywords "kw1,kw2" # only flag pages with these words in title/URL/web-crawl <url> --no-recurse # only fetch the start URL, no link following```## What it does1. **Fetches** the start URL (and optional follow-up pages) using the web fetch tool2. **Extracts** all hyperlinks (`<a href>`) from each page3. **Follows** links up to `--max-depth` or until `--max-pages` is reached4. **Flags** pages whose `<title>` or URL contains any of the `--bug-keywords` (default: `bug,issue,fix,todo,changelog`)5. **Writes** a CSV file `web-crawl-results.csv` in the current directory with columns: - `depth` — link hop number (0 = start URL) - `url` — the page URL - `title` — `<title>` tag content, or fallback `Untitled` - `bug-indicator` — `YES` if any bug keyword matched, `NO` otherwise - `source` — the URL where the link was discovered## How to use1. Invoke `/web-crawl https://example.com --max-pages 20 --bug-keywords "bug,issue,task"`2. The skill writes `web-crawl-results.csv` to the current working directory3. Open the CSV to review discovered pages; rows with `bug-indicator=YES` are likely bug-related4. Re-run with different keywords or a deeper max-depth to broaden/narrow the scan## Tips- External sites may block aggressive crawling; keep `--max-pages` moderate (20–50) and `--max-depth` low (2–3)- If a page requires JavaScript rendering, this skill cannot see it — it only reads static HTML- Use `--no-recurse` to just validate the start URL's title/links without following anything- The CSV can be imported into spreadsheets, databases, or issue trackers## Caveats- Only follows `<a href>` links; does not render CSS/JS or interact with forms- Respects robots.txt implicitly (no guarantee) — keep crawling responsible- Same-domain only: links pointing outside the start URL's host are ignored to avoid infinite cross-site crawling- If `--bug-keywords` match only the URL path (not the title), consider using more specific terms that appear in bug trackers## Crawling JavaScript SPAsStatic fetches return an empty shell for React/Vue/Angular sites (no links, nocontent - just a <div id=\"root\">). For those, use the companion Playwrightrenderer crawl_spa.py in this folder:```powershellpython crawl_spa.py https://corp.example.com ./example-crawl 30```- Requires pip install playwright && playwright install chromium- Renders each route headlessly, extracts main-content text, follows same-host links BFS, and saves one markdown file per page (with YAML frontmatter: title, url, captured_at) plus a manifest.json - ready for graphify ingestion.- Choose by site type: web-crawl.ps1 for server-rendered sites, crawl_spa.py when the first fetch returns an empty shell.

Frequently asked questions