Open-Web Acquisition — web_search / web_fetch
Two tools front the sandbox open-web engine (decepticon_sandbox.web):
web_search(query, provider="duckduckgo") — keyword OSINT over an
allowlisted search provider. Target-EXEMPT (it hits the provider, not the
target), so no engagement scope is required. Use it to DISCOVER: the org's
public footprint, exposed tech / version-specific advisories, leaked
references, doc/changelog/status pages, third-party subdomains and assets
named in the wild.
web_fetch(url, selector="", device="auto") — read ONE page's content,
escalating past WAF / anti-bot blocks. RoE target-gated (the url must be
in plan/roe.json scope) and SSRF-safe. Prefer it over curl-in-bash
whenever a public page is blocked, returns a challenge, or is JS-rendered.
Typical flow: web_search to find a URL → web_fetch to read it.
How web_fetch escalates (you do NOT drive this — the engine does)
web_fetch runs an escalation ladder inside the sandbox automatically:
- Phase 0 — official public APIs. Reddit / X(Twitter) / YouTube URLs are
auto-routed to their no-auth endpoints (Reddit
.rss, X tweet-result/oEmbed/
syndication, YouTube yt-dlp) BEFORE any WAF grid. Just pass the normal page
URL; a Phase-0 win shows verdict with profile=phase0:<platform>.
- curl_cffi TLS-impersonation grid. Real Safari/Chrome/Firefox JA3/JA4
fingerprints × URL transforms (mobile subdomain, …) × referer strategies,
tried exhaustively. This clears most WAFs (Cloudflare TLS, F5, AWS WAF, …).
- Headless browser tier. For JS challenges the curl grid can't clear
(Cloudflare Turnstile, Akamai Bot Manager, DataDome), a local headless
Chromium with stealth runs. There is no Playwright-MCP in the sandbox —
this local browser IS the final rung. If it still fails, that is a real wall.
You never pick a TLS profile, transform, or browser — the engine detects the WAF
and chooses. Your only knobs are selector and device.
R2 — HTTP 200 is NOT success
The engine validates every response (4-layer: challenge markers / body size /
cookie sensor / your selector). A 200 that is actually a challenge or empty
SPA shell is reported as a FAILURE, not content. Trust the tool's verdict /
[web_fetch OK|FAILED] line, never a raw status. Always pass a selector
(e.g. article, #content, [class*='product']) when you know the content
marker — it upgrades a "looks clean" guess into proven success.
R6 — a give-up is NOT exhaustion (the discipline that matters)
When web_fetch returns [web_fetch FAILED] it includes stop_reason= and
grid_exhausted=. Do not declare a page unreadable on the first FAILED.
- TERMINAL walls —
stop_reason is auth_required (login/paywall) or
not_found (404). These are real: record it and move on; retrying cannot
help.
- Everything else (
challenge, blocked, rate_limited, or a
non-exhausted grid) still has escalation left:
- rate_limited (429) is NOT terminal — back off, then retry.
- Retry once with
device="mobile" (mobile TLS + m. subdomain often
walk past a desktop block) and/or a precise selector.
- Only after a terminal
stop_reason may you honestly conclude the page is
unreachable.
This is the whole point of the engine: the cheap path failing is the start of
escalation, not permission to stop.
RoE & OSINT boundary
web_search is OSINT → not scope-gated (allowlisted provider egress only).
web_fetch is target-gated → the url must be in plan/roe.json scope; an
out-of-scope fetch returns ROE_REFUSED. SSRF-blocked (internal/metadata/
private) hosts are refused regardless.
- Reading public pages is OSINT, not exploitation. Use
bash/http_request
for ACTIVE probing of in-scope target infrastructure; use web_fetch to READ
pages (the target's own blocked/JS pages, or in-scope public assets).
Platform fast-paths (handled by Phase 0 — just pass the URL)
| Need |
Pass to web_fetch |
| Reddit thread/sub |
the reddit.com/... URL (auto .rss) |
| A specific tweet / X profile |
the x.com/.../status/... or profile URL |
| YouTube metadata / captions |
the youtube.com/watch?v=... URL (yt-dlp) |
| GitHub repo (metadata, stars, language) |
the github.com/<owner>/<repo> URL → repos API JSON |
| npm package (deps, versions, maintainers) |
the npmjs.com/package/<pkg> URL → registry JSON |
| PyPI package (versions, deps, urls) |
the pypi.org/project/<pkg> URL → JSON API |
| Any WAF-protected article/page |
the page URL — the grid + browser tier handle it |
GitHub/npm/PyPI return structured JSON (great for supply-chain / source recon)
instead of HTML — just pass the normal page URL and the engine routes it.
For keyword discovery on a platform (e.g. "what's said about X on Reddit"),
web_search("site:reddit.com <topic>") first, then web_fetch the result URLs.
1---2name: open-web3description: "Resilient public-page reading and OSINT keyword search — web_search (allowlisted-provider OSINT) and web_fetch (curl_cffi TLS-impersonation grid + headless-browser fallback that gets past WAF/403/anti-bot). Use when a fetch is blocked, a page is JS-rendered, or you need open-web OSINT about a target/org."4allowed-tools: Read5metadata:6 source: https://skillmd.com/skills/purpleailab/open-web7 subdomain: reconnaissance8 when_to_use: "blocked page, 403/402, WAF, cloudflare, akamai, datadome, captcha, JS-rendered page, open-web OSINT, read public page, reddit, x/twitter, youtube, search the web, third-party intel"9 tags: open-web, osint, web-fetch, web-search, waf-bypass10 upstream_ref: "adapted from fivetaku/insane-search (MIT) — SKILL.md R1–R7; decepticon_sandbox/web engine"11---1213# Open-Web Acquisition — web_search / web_fetch1415Two tools front the sandbox open-web engine (`decepticon_sandbox.web`):1617- **`web_search(query, provider="duckduckgo")`** — keyword OSINT over an18 allowlisted search provider. Target-EXEMPT (it hits the provider, not the19 target), so no engagement scope is required. Use it to DISCOVER: the org's20 public footprint, exposed tech / version-specific advisories, leaked21 references, doc/changelog/status pages, third-party subdomains and assets22 named in the wild.23- **`web_fetch(url, selector="", device="auto")`** — read ONE page's content,24 escalating past WAF / anti-bot blocks. **RoE target-gated** (the `url` must be25 in `plan/roe.json` scope) and SSRF-safe. Prefer it over `curl`-in-bash26 whenever a public page is blocked, returns a challenge, or is JS-rendered.2728Typical flow: **`web_search` to find a URL → `web_fetch` to read it.**2930## How web_fetch escalates (you do NOT drive this — the engine does)3132`web_fetch` runs an escalation ladder inside the sandbox automatically:33341. **Phase 0 — official public APIs.** Reddit / X(Twitter) / YouTube URLs are35 auto-routed to their no-auth endpoints (Reddit `.rss`, X tweet-result/oEmbed/36 syndication, YouTube `yt-dlp`) BEFORE any WAF grid. Just pass the normal page37 URL; a Phase-0 win shows `verdict` with `profile=phase0:<platform>`.382. **curl_cffi TLS-impersonation grid.** Real Safari/Chrome/Firefox JA3/JA439 fingerprints × URL transforms (mobile subdomain, …) × referer strategies,40 tried exhaustively. This clears most WAFs (Cloudflare TLS, F5, AWS WAF, …).413. **Headless browser tier.** For JS challenges the curl grid can't clear42 (Cloudflare Turnstile, Akamai Bot Manager, DataDome), a local headless43 Chromium with stealth runs. There is **no Playwright-MCP in the sandbox** —44 this local browser IS the final rung. If it still fails, that is a real wall.4546You never pick a TLS profile, transform, or browser — the engine detects the WAF47and chooses. Your only knobs are `selector` and `device`.4849## R2 — HTTP 200 is NOT success5051The engine validates every response (4-layer: challenge markers / body size /52cookie sensor / your `selector`). A 200 that is actually a challenge or empty53SPA shell is reported as a FAILURE, not content. Trust the tool's `verdict` /54`[web_fetch OK|FAILED]` line, never a raw status. **Always pass a `selector`**55(e.g. `article`, `#content`, `[class*='product']`) when you know the content56marker — it upgrades a "looks clean" guess into proven success.5758## R6 — a give-up is NOT exhaustion (the discipline that matters)5960When `web_fetch` returns `[web_fetch FAILED]` it includes `stop_reason=` and61`grid_exhausted=`. **Do not declare a page unreadable on the first FAILED.**6263- **TERMINAL walls** — `stop_reason` is `auth_required` (login/paywall) or64 `not_found` (404). These are real: record it and move on; retrying cannot65 help.66- **Everything else** (`challenge`, `blocked`, `rate_limited`, or a67 non-exhausted grid) still has escalation left:68 - **rate_limited (429) is NOT terminal** — back off, then retry.69 - Retry once with **`device="mobile"`** (mobile TLS + `m.` subdomain often70 walk past a desktop block) and/or a precise **`selector`**.71 - Only after a terminal `stop_reason` may you honestly conclude the page is72 unreachable.7374This is the whole point of the engine: the cheap path failing is the *start* of75escalation, not permission to stop.7677## RoE & OSINT boundary7879- `web_search` is OSINT → not scope-gated (allowlisted provider egress only).80- `web_fetch` is target-gated → the `url` must be in `plan/roe.json` scope; an81 out-of-scope fetch returns `ROE_REFUSED`. SSRF-blocked (internal/metadata/82 private) hosts are refused regardless.83- **Reading public pages is OSINT, not exploitation.** Use `bash`/`http_request`84 for ACTIVE probing of in-scope target infrastructure; use `web_fetch` to READ85 pages (the target's own blocked/JS pages, or in-scope public assets).8687## Platform fast-paths (handled by Phase 0 — just pass the URL)8889| Need | Pass to web_fetch |90|---|---|91| Reddit thread/sub | the `reddit.com/...` URL (auto `.rss`) |92| A specific tweet / X profile | the `x.com/.../status/...` or profile URL |93| YouTube metadata / captions | the `youtube.com/watch?v=...` URL (yt-dlp) |94| GitHub repo (metadata, stars, language) | the `github.com/<owner>/<repo>` URL → repos API JSON |95| npm package (deps, versions, maintainers) | the `npmjs.com/package/<pkg>` URL → registry JSON |96| PyPI package (versions, deps, urls) | the `pypi.org/project/<pkg>` URL → JSON API |97| Any WAF-protected article/page | the page URL — the grid + browser tier handle it |9899GitHub/npm/PyPI return structured JSON (great for supply-chain / source recon)100instead of HTML — just pass the normal page URL and the engine routes it.101102For keyword discovery on a platform (e.g. "what's said about X on Reddit"),103`web_search("site:reddit.com <topic>")` first, then `web_fetch` the result URLs.