Web Search Techniques

${PLUGIN_ROOT} is this plugin's install directory, the one that holds its plugin.json. If the host has not expanded it, resolve it from where this file was loaded.

Web Search Techniques

Shared knowledge base for research:quick-searcher, research:deep-researcher and the /research:team-research lead. Scope: web-only. Covers query formulation, source authority, tool usage, bot-block fallback, and anti-loop rules.

Query Formulation

Extract core concepts from the question before querying:

• Identify synonyms and domain terminology (e.g. "authentication": auth, login, signin, session, token, jwt, oauth, credentials)
• Account for abbreviations and full forms
• Add the year ("2026") when the query has temporal dependency
• Add "official" or "documentation" to push toward authoritative sources
• Quote exact phrases for precise matches
• Use site: to restrict to known-good domains (e.g. site:developer.mozilla.org)

Start broad, narrow progressively. Overly specific first queries miss adjacent information. Each refinement round incorporates terms surfaced in prior results.

Source Authority Ranking

Rank every source before citing:

1. Official documentation sites and API references: highest authority
2. RFC and specification documents: canonical for standards
3. GitHub issues, discussions, and source code: authoritative for specific libraries
4. Peer-reviewed or community-validated content: Stack Overflow with high votes, maintainers' blogs
5. General blog posts and tutorials: use only when nothing better exists
6. Deprioritize: SEO content farms, AI-generated summaries, scraped aggregators

Currency checks:

• Last modified date on the page
• Version numbers cited vs latest release
• Deprecation warnings

Search Backends

Two backends produce candidate pages. Neither produces claims: only a page that was read does.

Selection rule (the lead decides once per run and writes it into every spawn prompt): auto means serper when a key is available, else native. The chosen backend is stated in the plan, in each researcher report and in the final report header.

Where the key comes from, in order: SERPER_API_KEY in the environment, then ~/.serper_key. Ask the script rather than looking yourself: python ${PLUGIN_ROOT}/skills/web-search-techniques/scripts/websearch.py --check-key exits 0 when a key is available and 2 when none is, makes no network call, and costs no credit. A key the user pastes in chat is saved by piping it to --set-key, which is the lead's job at pre-flight and nobody else's: researchers never see a key, they call the script and the script reads it. Never echo a key, never write one into a prompt, a report or any other file, and never read the key file yourself.

A run never silently upgrades or degrades: with auto and no key the backend is native search and the plan says so, and a forced --backend serper with no key either collects one in chat or stops with the setup line, never quietly searches elsewhere.

What serper earns its call for:

• --vertical scholar for academic threads, --vertical news --since w|m for recency threads
• --num 30 to 50 on the orient round, to map a topic's vocabulary in one call (above 10 results costs 2 credits; the script says so on stderr)
• --since for "what changed lately" questions; before:YYYY-MM-DD / after:YYYY-MM-DD inside the query for exact windows
• Cross-index corroboration: a claim whose sources surface on both indexes ranks above one found on one index only

Operators both backends honour: site: (restrict to a domain), "exact phrase", -term (exclude), filetype:pdf, a year token (2026) for temporal queries, official or documentation to bias toward primary sources, version numbers when relevant (react 19).

Cost note (checked 2026-08-23): serper.dev gives 2,500 free queries, then $1.00 per 1,000 on the entry plan, down to $0.30 per 1,000 at volume; credits are deducted only on successful responses. A standard run makes on the order of 45-75 calls, a deep run 150-300. Tavily and Exa are LLM-oriented alternatives (LLM-ready snippets, neural search) that would slot behind the same script interface; not built.

Reading, Not Skimming

A search result is a candidate. A claim enters a researcher's ledger only from a page that was read:

1. WebFetch the page (prefer docs and primary sources; target anchors on long pages)
2. On a bot-block (403, 429, challenge page) or thin content (under ~200 useful characters), python3 ${PLUGIN_ROOT}/skills/web-search-techniques/scripts/webfetch.py <url>
3. If the playwright-skill plugin is installed and the page is a primary source the answer depends on, drive a real browser with it as the last resort; if it is not installed, record the URL under limitations and move on. This is a pointer, not a dependency.
4. Record: URL, title, the date the page carries, authority rank (below), and the claims taken from it

WebFetch Guidance

• Prefer documentation pages and API references over blog posts
• Evaluate fetched content: low-authority source means discard and re-search
• Large pages may be truncated: target specific sections (anchor URLs) when possible
• Track the accessed URL with date for citation

webfetch.py Fallback

When WebFetch returns a bot-block (403, 429, Cloudflare challenge) or thin content (under ~200 chars of useful text), fall back to the plugin's stealth fetcher:

python3 ${PLUGIN_ROOT}/skills/web-search-techniques/scripts/webfetch.py <url>

Behavior:

• Impersonates Chrome TLS fingerprint via curl_cffi
• Returns clean extracted text on stdout
• Exits 0 on success, 1 on timeout or error
• On failure, proceed without the result (do not retry in a loop)

Invocation options:

• --timeout SECONDS (default: 30)
• --max-chars CHARS (truncate output)
• --raw (return raw HTML instead of extracted text)

Requires Bash tool in the agent's tools: frontmatter.

Anti-Loop Rules

• Never repeat the exact same query or search parameters
• If a search returns nothing, change terminology, broaden the regex, or switch tool/target
• Maximum 2 failed attempts per sub-topic before pivoting or escalating
• After 2 failed attempts on the same angle, document the gap and proceed with what you have