Free Web Research
Use the bundled CLI to discover and read public sources without calling paid search or scraping APIs. Preserve evidence quality: prefer primary sources, check dates, and cite the final URLs.
Locate the CLI
Run the bundled wrapper from this Skill directory:
python3 scripts/free_web_research.py health
If it reports that the CLI is missing, tell the user to run python3 scripts/install.py from this Skill directory. The downloaded Skill contains the complete CLI source and creates an isolated environment. Do not install dependencies into the system Python ad hoc.
Route the request
Search local project files and .web-research-cache/ first.
For a known URL, fetch it directly:
python3 scripts/free_web_research.py fetch 'https://example.com/page'
For an unknown URL, use a zero-cost built-in Web Search available in the current harness when possible. Otherwise run DDGS/SearXNG:
python3 scripts/free_web_research.py search 'topic official source' --max-results 5
After discovering a URL, return to fetch; do not use a paid extractor by habit.
If direct extraction is incomplete, explicitly add free fallbacks:
python3 scripts/free_web_research.py fetch 'https://example.com/page' --fallback jina,browser
Consider Tavily or Firecrawl only when the JSON result contains escalation.recommended: true. Record the failed routes and expected paid call count before escalating. Never invoke a paid tool automatically from this Skill.
Read references/routing-policy.md before multi-source research, paid escalation, or changing cache policy. Read references/cli.md for all commands and output fields.
Search efficiently
- Start with one focused query and 3-5 results.
- Use
--domains for a strict allowlist and --prefer-domain to rank primary domains first.
- Use
--recency d|w|m|y only when freshness is required.
- Configure
FREE_WEB_RESEARCH_SEARXNG_URL to use a self-hosted SearXNG instance; otherwise auto uses DDGS.
- Treat DDGS/SearXNG results as discovery evidence. Fetch and inspect the source before making material claims.
Fetch safely
- Keep TLS verification enabled. Never add
--insecure, verify=False, credential harvesting, CAPTCHA bypass, or proxy rotation.
- Do not send private, authenticated, or confidential URLs through Jina Reader.
- Use Playwright only when it was installed with
--with-browser and direct extraction is inadequate.
- Respect access controls, robots policies, rate limits, copyright, and website terms.
- Keep realtime data TTL short; keep immutable filings and PDFs longer.
Handle outputs
Use JSON for agent workflows. Check ok, cache_hit, backend, quality_score, attempts, and escalation before relying on text or results. Use --format markdown only for human-readable output.
1---2name: web-search-free3description: Search the public web and fetch readable HTML, JSON, CSV, text, and PDF content with a local-first, zero-paid-API workflow. Use when Codex needs current web evidence, topic discovery, official-source lookup, URL extraction, PDF text, source collection, or a free fallback before Tavily/Firecrawl. Also use when users mention WebSearch, WebFetch, DDGS, SearXNG, Jina Reader, local scraping, web research cost control, or avoiding paid search credits.4---56# Free Web Research78Use the bundled CLI to discover and read public sources without calling paid search or scraping APIs. Preserve evidence quality: prefer primary sources, check dates, and cite the final URLs.910## Locate the CLI1112Run the bundled wrapper from this Skill directory:1314```bash15python3 scripts/free_web_research.py health16```1718If it reports that the CLI is missing, tell the user to run `python3 scripts/install.py` from this Skill directory. The downloaded Skill contains the complete CLI source and creates an isolated environment. Do not install dependencies into the system Python ad hoc.1920## Route the request21221. Search local project files and `.web-research-cache/` first.232. For a known URL, fetch it directly:2425 ```bash26 python3 scripts/free_web_research.py fetch 'https://example.com/page'27 ```28293. For an unknown URL, use a zero-cost built-in Web Search available in the current harness when possible. Otherwise run DDGS/SearXNG:3031 ```bash32 python3 scripts/free_web_research.py search 'topic official source' --max-results 533 ```34354. After discovering a URL, return to `fetch`; do not use a paid extractor by habit.365. If direct extraction is incomplete, explicitly add free fallbacks:3738 ```bash39 python3 scripts/free_web_research.py fetch 'https://example.com/page' --fallback jina,browser40 ```41426. Consider Tavily or Firecrawl only when the JSON result contains `escalation.recommended: true`. Record the failed routes and expected paid call count before escalating. Never invoke a paid tool automatically from this Skill.4344Read [references/routing-policy.md](references/routing-policy.md) before multi-source research, paid escalation, or changing cache policy. Read [references/cli.md](references/cli.md) for all commands and output fields.4546## Search efficiently4748- Start with one focused query and 3-5 results.49- Use `--domains` for a strict allowlist and `--prefer-domain` to rank primary domains first.50- Use `--recency d|w|m|y` only when freshness is required.51- Configure `FREE_WEB_RESEARCH_SEARXNG_URL` to use a self-hosted SearXNG instance; otherwise `auto` uses DDGS.52- Treat DDGS/SearXNG results as discovery evidence. Fetch and inspect the source before making material claims.5354## Fetch safely5556- Keep TLS verification enabled. Never add `--insecure`, `verify=False`, credential harvesting, CAPTCHA bypass, or proxy rotation.57- Do not send private, authenticated, or confidential URLs through Jina Reader.58- Use Playwright only when it was installed with `--with-browser` and direct extraction is inadequate.59- Respect access controls, robots policies, rate limits, copyright, and website terms.60- Keep realtime data TTL short; keep immutable filings and PDFs longer.6162## Handle outputs6364Use JSON for agent workflows. Check `ok`, `cache_hit`, `backend`, `quality_score`, `attempts`, and `escalation` before relying on `text` or `results`. Use `--format markdown` only for human-readable output.
qiran87