# Web Crawler

> Web Crawler Workflow

- Skill: `lingbol088-spec/web-crawler` (Agent Skill)
- Install (CLI): `npx skillmds@latest add lingbol088-spec/web-crawler`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lingbol088-spec/web-crawler/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: lingbol088-spec (https://skillmd.com/u/lingbol088-spec)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lingbol088-spec/web-crawler

---


# Web Crawler Workflow

Auto-chain without confirmation:

Phase 1 — Discover:
  Spider the target domain. Extract all links, forms, endpoints, JS files.
  Identify: login pages, admin panels, API endpoints, upload forms.

Phase 2 — Fingerprint:
  Tech stack: server, framework, CMS, JS libraries, CDN.
  Headers: security headers missing, cookie flags, CORS config.

Phase 3 — Attack surface:
  `skills/web-pentest/scripts/sqli_scanner.py` → all form/query params
  `skills/web-pentest/scripts/xss_fuzzer.py` → all reflected inputs
  `skills/web-pentest/scripts/auth_bypass.py` → login pages, JWT tokens

Phase 4 — Deep scan:
  Directory brute-force on discovered paths.
  API endpoint fuzzing for hidden endpoints.
  File upload testing on discovered upload forms.

Phase 5 — Data extraction:
  All discovered data written to disk.
  Structured JSON output: {urls, forms, endpoints, vulns, tech}

Chat shows: [OK] → $PATH/crawl-results.json