dreamlab-ai Web Researcher Skill
Production-grade live-web research bridge: search across the engine you choose, narrow to the sources you trust (lenses), read the full source (not snippets), and get citations you can verify — never fabricated, never a closed pre-synthesized garden. Runs locally/private; never invokes a browser from inside this skill.
Which web-search skill? (read this first — avoids the common mix-up)
Search priority: 1. ceramic → 2. perplexity → 3. Claude WebSearch
Complex/important? → all three in parallel
Verifiable citations needed? → web-researcher (this)
| Need | Skill |
|---|---|
| Default web search — keyword, rich 8k extracts, fast | ceramic-search (primary) |
| Authoritative sources, synthesized answer, academic/policy filters | perplexity-research (secondary) |
| Quick built-in fallback | Claude WebSearch (tertiary) |
| Verifiable, reputation-attached research — real citations, full sources, you control which domains are searched (lenses), private/local; client work, filings, publications, medical/legal/finance | web-researcher (this) |
| Multi-agent deep report — fan-out + adversarial verification + cited synthesis (orchestrates searchers; can use any of the above as a backend) | deep-research |
| Expand a single known URL | scrape_page (this skill, full text/PDF/YouTube) ; JS-rendered → browser sidecar |
| Interactive browser (login/click/JS render) | browser / playwright |
One-liner: ceramic = primary keyword search with dense extracts; perplexity = secondary, synthesized + authoritative; web-researcher = verifiable citations + lenses; deep-research = the harness over all.
When To Use
- Research where your reputation is on the line — citations must be real and checkable.
- Restrict search to trusted sources via lenses (PubMed/arXiv/SEC/.gov, not random blogs).
- Read full articles — web pages, PDF/DOCX/PPTX, YouTube transcripts, Hacker News threads.
- Academic work: real papers + DOIs (
academic_search), citation neighborhoods (citation_graph). - Domain search: courts (
legal_search), trials (clinical_search), macro/econ (econ_search). - Verify a citation before relying on it (
verify_citation) or audit a whole reference list (audit_bibliography); snapshot a source to Wayback (archive_source); export a bibliography (format_bibliography). - Curated resource discovery (
awesome_list_search) or a structured company/brand pass (brand_research). - Session-tracked multi-step digs (
sequential_search) with recovery + export.
When Not To Use
- Quick casual lookup, no citing needed →
perplexity-researchor Claude built-in search. - Single known URL summary →
scrape_page(this skill); JS-rendered →browsersidecar. - Interactive browser flows (login, click, form-fill) →
browser/playwright. - WebGPU/WebGL rendering validation →
browser/chrome-cdp. - Multi-agent report with adversarial verification →
deep-research(it can call this skill). - Iterative metric-optimisation experiment loops →
autoresearch. - Grounding in OUR formal ontology/KG →
ontology-augment.
Connection
MCP stdio server registered as web-researcher in mcp/mcp.json. Deployed binary
is web-researcher-mcp (Nix-baked, resolved from $PATH; see the frontmatter
version field for the exact pin); the full toolset — the 21 tools tabled below —
is registered and live, verified against the running server's tools/list.
Registration, version-bump (flake.nix pin) and provider-gating detail:
references/deployment.md.
Tools (live set depends on configured providers — see notes)
Search & read
| Tool | What it does |
|---|---|
web_search |
Search the web; optionally restrict to trusted sources via a lens |
search_and_scrape |
Search then read the best results, quality-scored + deduped |
scrape_page |
Read any URL in full — pages, PDF/DOCX/PPTX, YouTube transcripts, HN (API); mode:raw for verbatim |
image_search |
Images by size/type/colour/format |
news_search |
Recent news with date controls + source filtering |
sequential_search |
Multi-step research that remembers prior findings |
awesome_list_search |
Mine curated GitHub "awesome-*" lists for a topic's canonical resources |
brand_research |
Structured research pass on a company/brand |
Domain search
| Tool | Source |
|---|---|
academic_search |
Real papers + DOIs (OpenAlex/Semantic Scholar/arXiv/PubMed/IEEE) |
citation_graph |
Walk a paper's citing/cited neighborhood — needs a citation-capable provider |
patent_search |
US/EP/WO/JP/CN/KR patent offices + classification |
legal_search |
US court opinions/dockets via CourtListener |
clinical_search |
ClinicalTrials.gov (discovery, not medical advice) |
econ_search |
World Bank indicators (keyless) + FRED US macro series |
Citation integrity & output
| Tool | What it does |
|---|---|
verify_citation |
Does a citation exist, match a real record, is it retracted/dead-link? Evidence, not a verdict |
audit_bibliography |
Audit a whole CSL-JSON/RIS/BibTeX list/session — per-entry + corpus flags |
verify_recommendation |
Check a recommended source before relying on it |
archive_source |
Capture a fresh Wayback snapshot so a cited page stays verifiable (write tool) |
format_bibliography |
APA / MLA / BibTeX / RIS / CSL-JSON (Zotero/EndNote/Mendeley-ready) |
Session / memory
| Tool | What it does |
|---|---|
get_research_session |
Recover a session after context loss |
research_export |
Export a provenance-tracked report (md/JSON) |
All 21 tools above are live in the deployed binary (verified via tools/list; see
the frontmatter version field for the exact pin). Some are inert without provider
keys — see references/deployment.md.
docs/TOOLS.md upstream is the authoritative, CI-verified tool list + schemas.
Search Lenses (the differentiator)
Lenses restrict results to a curated set of trusted domains for a field, instead of the
whole web. Built-in lenses ship with the binary (catalog at lenses://catalog):
academic, academic-extended, clinical, legal, finance, government,
journalism, devops, docs (+ custom JSON lenses). Reference by short name:
web_search({ query: "ML-KEM constant-time implementations", lens: "academic", count: 8 })
web_search({ query: "FDA breakthrough designation 2026", lens: "clinical" })
MCP Resources & Prompts
Live status/diagnostics the agent can read: stats://tools, stats://sessions,
stats://rate-limits, stats://providers, lenses://catalog,
diagnostics://errors/recent, diagnostics://health, and a large-payload artifact
store research://artifact/{id}. Ready-made research prompt templates appear as / commands.
Browser Delegation (Critical)
web-researcher-mcp's tier-4 scrape (go-rod + stealth) would auto-download its own
Chromium per pod. In agentbox this is disabled — a second Chromium duplicates the
browsercontainer sidecar (Chrome Beta 149+, NVIDIA Vulkan, VNC :5903, MCP SSE :8931):
SCRAPER_DISABLE_BROWSER=true
CHROME_PATH= # empty -> tier 4 hard-fails fast
When tiers 1–3 (markdown negotiation, stealth HTTP, HTML parser) can't extract a
JS-rendered page (NEEDS_BROWSER), switch to the browser skill — do not retry inside this MCP:
1. scrape_page(url) -> tiers 1-3 ok => done ; all fail => NEEDS_BROWSER
2. browser_navigate({url}) on browser-gpu
3. browser_snapshot() / browser_evaluate(...)
One Chrome surface, one GPU allocation, one audit trail.
Required Configuration
Set at least one search provider (read from session env):
GOOGLE_CUSTOM_SEARCH_API_KEY GOOGLE_CUSTOM_SEARCH_ID # PSE (default)
BRAVE_API_KEY # Brave
SERPER_API_KEY # Serper.dev
SEARCHAPI_API_KEY # SearchAPI.io
EXA_API_KEY # Exa (neural search + citation-graph provider)
SEARXNG_URL # self-hosted
Multi-provider routing with per-provider circuit breakers + failover:
export SEARCH_ROUTING=brave,google,serper
# or per-operation JSON:
export SEARCH_ROUTING='{"web":"brave,google","news":"brave,serper","images":"google,brave","default":"brave,google"}'
Composition Patterns
| Goal | Recipe |
|---|---|
| Deep cited report | web-researcher (search/verify) → deep-research (parallel agents + adversarial verify) |
| Trustworthy academic claim | academic_search → verify_citation → citation_graph → format_bibliography |
| Keep a cited source alive | archive_source (Wayback) before publishing |
| Fast casual answer | perplexity-research instead |
| Single known URL | scrape_page (this skill); JS-rendered → browser sidecar |
| JS-rendered SPA | scrape_page → fallback to browser skill |
| Add provenance sidecars to a deliverable | pair with provenance-tracking |
Health Check
Hold stdin open long enough for the response (the server exits on EOF):
{ printf '%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"c","version":"1"}}}' \
'{"jsonrpc":"2.0","method":"notifications/initialized"}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}'; sleep 3; } \
| web-researcher-mcp 2>/dev/null | grep -o '"name":"[a-z_]*"' | sort -u
References
- Deployment / Nix pin / provider gating:
references/deployment.md - Upstream: https://github.com/zoharbabin/web-researcher-mcp
- Authoritative tools:
docs/TOOLS.md; deployment:docs/DEPLOYMENT.md(upstream) - Related skills:
perplexity-research(fast closed synthesis),deep-research(multi-agent harness),browser(JS-rendered pages / interactive flows),provenance-tracking,autoresearch,ontology-augment