# Web Search Techniques > Knowledge base for web research: query formulation, source authority ranking, the two search backends (native WebSearch, optional serper.dev through websearch.py), reading pages with WebFetch and the webfetch.py bot-block fallback, and the anti-loop rules. Used by quick-searcher, deep-researcher and /research:team-research. TRIGGER WHEN: performing web research with WebSearch, WebFetch, or the research plugin's scripts. DO NOT TRIGGER WHEN: searching a local codebase (use Grep or Glob directly). - Skill: `acaprino/web-search-techniques-3` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add acaprino/web-search-techniques-3` - Raw SKILL.md: https://api.skillmd.com/api/skills/acaprino/web-search-techniques-3/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Research & Search - Author: acaprino (https://skillmd.com/u/acaprino) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/acaprino/web-search-techniques-3 --- > `` names the directory that holds this plugin's `.codex-plugin/plugin.json`. Resolve it once from where this file was loaded, then substitute it into every path below that starts with it. # Web Search Techniques Shared knowledge base for `research:quick-searcher`, `research:deep-researcher` and the `/research:team-research` lead. Scope: web-only. Covers query formulation, source authority, tool usage, bot-block fallback, and anti-loop rules. ## Query Formulation Extract core concepts from the question before querying: - Identify synonyms and domain terminology (e.g. "authentication": auth, login, signin, session, token, jwt, oauth, credentials) - Account for abbreviations and full forms - Add the year ("2026") when the query has temporal dependency - Add "official" or "documentation" to push toward authoritative sources - Quote exact phrases for precise matches - Use `site:` to restrict to known-good domains (e.g. `site:developer.mozilla.org`) Start broad, narrow progressively. Overly specific first queries miss adjacent information. Each refinement round incorporates terms surfaced in prior results. ## Source Authority Ranking Rank every source before citing: 1. **Official documentation sites and API references**: highest authority 2. **RFC and specification documents**: canonical for standards 3. **GitHub issues, discussions, and source code**: authoritative for specific libraries 4. **Peer-reviewed or community-validated content**: Stack Overflow with high votes, maintainers' blogs 5. **General blog posts and tutorials**: use only when nothing better exists 6. **Deprioritize**: SEO content farms, AI-generated summaries, scraped aggregators Currency checks: - Last modified date on the page - Version numbers cited vs latest release - Deprecation warnings ## Search Backends Two backends produce candidate pages. Neither produces claims: only a page that was read does. | Backend | When | How | |---|---|---| | Native `WebSearch` | Default. Always available when the tool is in the toolset | The tool call, with the operators below | | serper.dev (Google) | A key is available, or `--backend serper` | `python /skills/web-search-techniques/scripts/websearch.py "" [--vertical search|news|scholar] [--num N] [--since h|d|w|m|y] [--gl CC] [--hl LANG] [--page P] [--json]` via Bash | Selection rule (the lead decides once per run and writes it into every spawn prompt): `auto` means serper when a key is available, else native. The chosen backend is stated in the plan, in each researcher report and in the final report header. Where the key comes from, in order: `SERPER_API_KEY` in the environment, then `~/.serper_key`. Ask the script rather than looking yourself: `python /skills/web-search-techniques/scripts/websearch.py --check-key` exits 0 when a key is available and 2 when none is, makes no network call, and costs no credit. A key the user pastes in chat is saved by piping it to `--set-key`, which is the lead's job at pre-flight and nobody else's: researchers never see a key, they call the script and the script reads it. Never echo a key, never write one into a prompt, a report or any other file, and never read the key file yourself. A run never silently upgrades or degrades: with `auto` and no key the backend is native search and the plan says so, and a forced `--backend serper` with no key either collects one in chat or stops with the setup line, never quietly searches elsewhere. What serper earns its call for: - `--vertical scholar` for academic threads, `--vertical news --since w|m` for recency threads - `--num 30` to `50` on the orient round, to map a topic's vocabulary in one call (above 10 results costs 2 credits; the script says so on stderr) - `--since` for "what changed lately" questions; `before:YYYY-MM-DD` / `after:YYYY-MM-DD` inside the query for exact windows - Cross-index corroboration: a claim whose sources surface on both indexes ranks above one found on one index only Operators both backends honour: `site:` (restrict to a domain), `"exact phrase"`, `-term` (exclude), `filetype:pdf`, a year token (`2026`) for temporal queries, `official` or `documentation` to bias toward primary sources, version numbers when relevant (`react 19`). Cost note (checked 2026-08-23): serper.dev gives 2,500 free queries, then $1.00 per 1,000 on the entry plan, down to $0.30 per 1,000 at volume; credits are deducted only on successful responses. A `standard` run makes on the order of 45-75 calls, a `deep` run 150-300. Tavily and Exa are LLM-oriented alternatives (LLM-ready snippets, neural search) that would slot behind the same script interface; not built. ## Reading, Not Skimming A search result is a candidate. A claim enters a researcher's ledger only from a page that was read: 1. `WebFetch` the page (prefer docs and primary sources; target anchors on long pages) 2. On a bot-block (403, 429, challenge page) or thin content (under ~200 useful characters), `python3 /skills/web-search-techniques/scripts/webfetch.py ` 3. If the `playwright-skill` plugin is installed and the page is a primary source the answer depends on, drive a real browser with it as the last resort; if it is not installed, record the URL under limitations and move on. This is a pointer, not a dependency. 4. Record: URL, title, the date the page carries, authority rank (below), and the claims taken from it ## WebFetch Guidance - Prefer documentation pages and API references over blog posts - Evaluate fetched content: low-authority source means discard and re-search - Large pages may be truncated: target specific sections (anchor URLs) when possible - Track the accessed URL with date for citation ## webfetch.py Fallback When WebFetch returns a bot-block (403, 429, Cloudflare challenge) or thin content (under ~200 chars of useful text), fall back to the plugin's stealth fetcher: ```bash python3 /skills/web-search-techniques/scripts/webfetch.py ``` Behavior: - Impersonates Chrome TLS fingerprint via curl_cffi - Returns clean extracted text on stdout - Exits 0 on success, 1 on timeout or error - On failure, proceed without the result (do not retry in a loop) Invocation options: - `--timeout SECONDS` (default: 30) - `--max-chars CHARS` (truncate output) - `--raw` (return raw HTML instead of extracted text) Requires `Bash` tool in the agent's `tools:` frontmatter. ## Anti-Loop Rules - Never repeat the exact same query or search parameters - If a search returns nothing, change terminology, broaden the regex, or switch tool/target - Maximum 2 failed attempts per sub-topic before pivoting or escalating - After 2 failed attempts on the same angle, document the gap and proceed with what you have