Terminal Web Research

Terminal-Only Web Research

Source-grounded web research using only curl — for subagents/sandboxes where web_search/web_extract are not configured, or when configured search tools fail. Validated 2026-08-11 on a podcast-research sweep (macOS, curl 8.x).

When to Use

• You are a research subagent and web_search/web_extract are not in your toolset.
• Search tools exist but error out (subscription/rate-limit/blocked), and browser tools are unavailable.
• You must verify sources, quotes, or URLs for a source-grounded deliverable and need exact article URLs, not just snippets.

Core rule

Search engines block curl; article pages usually do not. Once you have even one candidate URL, stop searching and fetch the article directly with a desktop browser UA. Direct fetches of nymag.com, usatoday.com, theguardian.com, rollingstone.com, substack all succeeded while every search engine challenged the same IP.

Use a full desktop UA, e.g.: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

Search-engine fallback chain (tested with curl)

Brave raw-HTML parse (no JS):

import re, html
raw = <curl stdout>
links = []
for m in re.finditer(r'<a[^>]+href="(https?://[^"]+)"', raw):
    u = html.unescape(m.group(1))
    if any(x in u for x in ("brave.com", "braveusercontent", "reddit.com/", "youtube.com")):
        continue
    if u not in links: links.append(u)
snips = [re.sub(r"<[^>]+>", "", s).strip()[:200] for s in
         re.findall(r'<div class="snippet[^"]*"[^>]*>(.*?)</div>', raw, flags=re.S)]

Notes: DDG blocking is transient — an earlier run from the same IP produced clean results; retry after 20-60s or switch to Brave. Sleep 2-3s between queries.

URL discovery without a search engine

• Wikipedia ?action=raw — curl 'https://en.wikipedia.org/wiki/<Title>?action=raw' returns wikitext with ALL citation URLs intact. Rendered HTML snapshots strip citation URLs into bare [5] footnotes — useless. This recovered an exact NYMag article URL after two guessed URL patterns 404'd.
• Wayback CDX API — find a real article URL from a known site-path prefix: https://web.archive.org/cdx/search/cdx?url=<site>/<path>/*&output=text&limit=500&collapse=urlkey then grep. &filter=urlkey:.*<term>.* works on small sets.
  • Do NOT use domain-wide matchType=domain (504 timeouts).
  • archive.org/wayback/available?url=... rate-limits fast (429); ≥20s between calls.
• Decode saved DDG result pages — DDG result links are redirects //duckduckgo.com/l/?uddg=<urlencoded-target>; recover targets with urllib.parse.unquote(re.search(r'uddg=([^&]+)', href).group(1)).

Fetching & block detection

• Bulk-fetch many URLs in ONE script (ThreadPoolExecutor 5-8 workers, 3 retries, one curl subprocess per URL). See scripts/fetch_strip_pages.py for a ready-to-run fetcher+stripper (reads a slug→URL mapping, saves pages/<slug>.html + pages/<slug>.txt, retries, treats <20KB as failure).
• Size thresholds: real article pages were 100KB-1.8MB; Cloudflare/404/ challenge shells were ~5-15KB. < 20KB ⇒ suspected block. Some outlets (Consumer Reports, Zendesk help pages, radiologybusiness.com, mediaite.com) return ~6KB JS-only shells even for correct URLs — stripped text of ~60 chars = blocked/JS-only; find the story syndicated elsewhere (yahoo.com, dnyuz.com).
• Strip recipe: remove <script>/<style> (flags=re.S), remove tags, html.unescape, collapse whitespace.
• Paywalls: article body truncates at the paywall even when the page fetches fully (NYMag cut mid-sentence). Don't read linearly — str.find() on key terms (names, dates, "spokesperson", "denies") and print ±1000-char windows.

Pitfalls

• Tool-call budget: a research sweep easily hits a 50-call ceiling mid-write. Batch ALL fetches into one script call, run searches via one script, and write the deliverable file incrementally as sections finish — never defer the entire write to the end.
• Don't trust memory of URLs/titles: verify each URL (guessed URL patterns failed; CDX/action=raw found the real ones).
• Frame transient blocks as retry-or-fallback, never "engine X is broken" — a later run from the same IP may succeed.
• Attribution discipline: for each claim record outlet + author + date + access date; quoted spokespeople inside an article are the subject's response, not the outlet's position.