Web Search Techniques

Web Search Techniques

Shared knowledge base for research:quick-searcher, research:deep-researcher and the /research:team-research lead. Scope: web-only. Covers query formulation, source authority, tool usage, bot-block fallback, and anti-loop rules.

Query Formulation

Extract core concepts from the question before querying:

• Identify synonyms and domain terminology (e.g. "authentication": auth, login, signin, session, token, jwt, oauth, credentials)
• Account for abbreviations and full forms
• Add the year ("2026") when the query has temporal dependency
• Add "official" or "documentation" to push toward authoritative sources
• Quote exact phrases for precise matches
• Use site: to restrict to known-good domains (e.g. site:developer.mozilla.org)

Start broad, narrow progressively. Overly specific first queries miss adjacent information. Each refinement round incorporates terms surfaced in prior results.

Source Authority Ranking

Rank every source before citing:

1. Official documentation sites and API references: highest authority
2. RFC and specification documents: canonical for standards
3. GitHub issues, discussions, and source code: authoritative for specific libraries
4. Peer-reviewed or community-validated content: Stack Overflow with high votes, maintainers' blogs
5. General blog posts and tutorials: use only when nothing better exists
6. Deprioritize: SEO content farms, AI-generated summaries, scraped aggregators

Currency checks:

• Last modified date on the page
• Version numbers cited vs latest release
• Deprecation warnings

Search Backends

Two backends produce candidate pages. Neither produces claims: only a page that was read does.

Selection rule (the lead decides once per run and writes it into every spawn prompt): auto means serper when a key is available, else native. The chosen backend is stated in the plan, in each researcher report and in the final report header.

Where the key comes from, in order: SERPER_API_KEY in the environment, then ~/.serper_key. Ask the script rather than looking yourself: python ${CLAUDE_PLUGIN_ROOT}/skills/web-search-techniques/scripts/websearch.py --check-key exits 0 when a key is available and 2 when none is, makes no network call, and costs no credit. A key the user pastes in chat is saved by piping it to --set-key, which is the lead's job at pre-flight and nobody else's: researchers never see a key, they call the script and the script reads it. Never echo a key, never write one into a prompt, a report or any other file, and never read the key file yourself.

A run never silently upgrades or degrades: with auto and no key the backend is native search and the plan says so, and a forced --backend serper with no key either collects one in chat or stops with the setup line, never quietly searches elsewhere.

What serper earns its call for:

• --vertical scholar for academic threads, --vertical news --since w|m for recency threads
• --num 30 to 50 on the orient round, to map a topic's vocabulary in one call (above 10 results costs 2 credits; the script says so on stderr)
• --since for "what changed lately" questions; before:YYYY-MM-DD / after:YYYY-MM-DD inside the query for exact windows
• Cross-index corroboration: a claim whose sources surface on both indexes ranks above one found on one index only

Operators both backends honour: site: (restrict to a domain), "exact phrase", -term (exclude), filetype:pdf, a year token (2026) for temporal queries, official or documentation to bias toward primary sources, version numbers when relevant (react 19).

Cost note (checked 2026-08-23): serper.dev gives 2,500 free queries, then $1.00 per 1,000 on the entry plan, down to $0.30 per 1,000 at volume; credits are deducted only on successful responses. A standard run makes on the order of 45-75 calls, a deep run 150-300. Tavily and Exa are LLM-oriented alternatives (LLM-ready snippets, neural search) that would slot behind the same script interface; not built.

Reading, Not Skimming

A search result is a candidate. A claim enters a researcher's ledger only from a page that was read:

1. WebFetch the page (prefer docs and primary sources; target anchors on long pages)
2. On a bot-block (403, 429, challenge page) or thin content (under ~200 useful characters), python3 ${CLAUDE_PLUGIN_ROOT}/skills/web-search-techniques/scripts/webfetch.py <url>
3. If the playwright-skill plugin is installed and the page is a primary source the answer depends on, drive a real browser with it as the last resort; if it is not installed, record the URL under limitations and move on. This is a pointer, not a dependency.
4. Record: URL, title, the date the page carries, authority rank (below), and the claims taken from it

WebFetch Guidance

• Prefer documentation pages and API references over blog posts
• Evaluate fetched content: low-authority source means discard and re-search
• Large pages may be truncated: target specific sections (anchor URLs) when possible
• Track the accessed URL with date for citation

webfetch.py Fallback

When WebFetch returns a bot-block (403, 429, Cloudflare challenge) or thin content (under ~200 chars of useful text), fall back to the plugin's stealth fetcher:

python3 ${CLAUDE_PLUGIN_ROOT}/skills/web-search-techniques/scripts/webfetch.py <url>

Behavior:

• Impersonates Chrome TLS fingerprint via curl_cffi
• Returns clean extracted text on stdout
• Exits 0 on success, 1 on timeout or error
• On failure, proceed without the result (do not retry in a loop)

Invocation options:

• --timeout SECONDS (default: 30)
• --max-chars CHARS (truncate output)
• --raw (return raw HTML instead of extracted text)

Requires Bash tool in the agent's tools: frontmatter.

Anti-Loop Rules

• Never repeat the exact same query or search parameters
• If a search returns nothing, change terminology, broaden the regex, or switch tool/target
• Maximum 2 failed attempts per sub-topic before pivoting or escalating
• After 2 failed attempts on the same angle, document the gap and proceed with what you have