Deep Research

Deep Research

Performs iterative web research via local SearXNG (Google-free, VPN-routed).

Quick Usage

python3 ~/.clawdbot/skills/deep-research/deep_research.py "your research question"

Or use the CLI wrapper:

deep-research "what are the best practices for kubernetes security in 2026"

How It Works

1. Iterative Search - Up to 5 iterations with query refinement
2. Content Fetching - Scrapes full page content from valid URLs
3. Deduplication - Tracks seen URLs to avoid duplicates
4. Report Generation - Produces markdown with citations

Algorithm

for iteration in 1..5:
    query = refine_query(original_query, iteration)
    results = search_searxng(query, offset=iteration * 10)

    for result in results:
        if url not in seen_urls and domain not in ignored:
            content = fetch_and_scrape(url)
            add_to_findings(title, url, content)

    if sufficient_results:
        break

generate_markdown_report(findings, citations)

Query Refinement

Each iteration adds context terms:

• Iteration 1: Original query
• Iteration 2: + "detailed analysis"
• Iteration 3: + "comprehensive guide"
• Iteration 4: + "in-depth review"
• Iteration 5: + "research findings"

Configuration

Edit ~/.clawdbot/skills/deep-research/deep_research.py:

SEARXNG_URL = "http://localhost:8888"  # Your SearXNG instance
MAX_ITERATIONS = 5                      # Search iterations
RESULTS_PER_PAGE = 10                   # Results per iteration
PAGE_CONTENT_LIMIT = 2000               # Max words per source
REQUEST_TIMEOUT = 20                    # Fetch timeout (seconds)

Ignored Domains

Social media and low-value domains are excluded:

• youtube.com, facebook.com, twitter.com
• instagram.com, tiktok.com, pinterest.com, linkedin.com

Edit IGNORED_DOMAINS list to customize.

Output Format

# Deep Research Report

**Query:** your research question
**Date:** 2026-01-27 15:30
**Sources:** 8

---

## Research Findings

### [1] Article Title
**Source:** https://example.com/article

Content preview from the article...

### [2] Another Source
...

---

## Sources

1. [Article Title](https://example.com/article)
2. [Another Source](https://example.com/other)

Privacy Features

• No Google/Bing - Uses privacy-respecting engines only
• VPN Routed - Traffic goes through Tailscale/Mullvad
• Local Processing - All synthesis happens locally
• No API Keys - Self-hosted SearXNG, no external dependencies

Requirements

• Python 3.8+
• Local SearXNG instance at port 8888
• Python packages: aiohttp, beautifulsoup4

Install dependencies:

pip install aiohttp beautifulsoup4

Examples

Research a technical topic:

python3 ~/.clawdbot/skills/deep-research/deep_research.py "rust async runtime comparison tokio vs async-std 2026"

Investigate a concept:

python3 ~/.clawdbot/skills/deep-research/deep_research.py "zero knowledge proofs practical applications"

Compare technologies:

python3 ~/.clawdbot/skills/deep-research/deep_research.py "comparing vector databases pinecone vs milvus vs qdrant"

Troubleshooting

No results found:

• Check SearXNG is running: curl http://localhost:8888
• Verify query isn't too specific
• Try broader search terms

Slow performance:

• Reduce MAX_ITERATIONS
• Decrease RESULTS_PER_PAGE
• Some sites have rate limiting

Content not extracted:

• Site may require JavaScript (not supported)
• Try the URL directly in browser
• Content may be behind paywall