# Fetch Web Intel

> Use when Codex must fetch web pages, scrape selectors, or crawl sitemaps through the remote Fetch/Web MCP server catalog entry.

- Skill: `majiayu000/fetch-web-intel` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add majiayu000/fetch-web-intel`
- Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/fetch-web-intel/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: majiayu000 (https://skillmd.com/u/majiayu000)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/majiayu000/fetch-web-intel

---


# Fetch Web Intel

## Purpose
Plan and execute web data collection tasks via the hosted Fetch MCP server registered in `servers/fetch`. Ideal for capturing article summaries, metadata, or targeted DOM fragments.

## Setup Checklist
1. Verify `FETCH_MCP_ENDPOINT` / `FETCH_MCP_API_KEY` exist and the endpoint is listed in `mcp.json`.
2. Confirm the remote server’s allow list includes the target domains; update the provider portal if not.
3. Decide on user agent overrides when sites block generic bots.

## Workflow
1. **Scope** – list URLs plus extraction goals (full page HTML, CSS selector, sitemap traversal). Batch by domain to leverage connection reuse.
2. **Execute** – call the MCP `fetch`, `scrape`, or `search` tools. Supply timeout hints for heavier pages.
3. **Normalize** – clean the payload (strip scripts, collapse whitespace) before storing or handing to other skills.
4. **Log** – capture HTTP status and rate-limit headers so we can backoff or retry intelligently.

## Notes
- Respect robots: if the server returns `blockedByRobots`, inform the user instead of retrying.
- Cap parallel requests to 3 per domain to avoid provider throttling.
- Store extracted outputs under `docs/intel/` through the filesystem skill if they need persistence beyond the current run.