Web Research

Web Research

Overview

Conduct thorough web research using the Brave Search API. Search for information, fetch and extract content from web pages, cross-reference multiple sources, and compile structured research reports with proper citations. Designed for deep-dive research that goes beyond a single search query.

Instructions

When a user asks you to research a topic on the web, follow these steps:

Step 1: Plan the research

Before searching, define a research plan:

• Central question: What is the user trying to learn or decide?
• Sub-questions: Break the topic into 3-5 specific queries
• Source types: What kinds of sources are valuable (academic, news, official docs, forums)?
• Scope: How deep should the research go?

Example plan for "Is Rust ready for production web development?":

Central question: Can Rust be used for production web APIs today?

Sub-queries:
  1. "Rust web frameworks comparison 2025"
  2. "Rust production web services case studies"
  3. "Rust vs Go web performance benchmarks"
  4. "Rust web development challenges limitations"
  5. "companies using Rust in production backend"

Step 2: Search with Brave Search API

import requests
import os

BRAVE_API_KEY = os.environ.get("BRAVE_API_KEY")
BRAVE_SEARCH_URL = "https://api.search.brave.com/res/v1/web/search"

def brave_search(query, count=10, freshness=None):
    """Search the web using Brave Search API.

    Args:
        query: Search query string
        count: Number of results (max 20)
        freshness: Optional filter: 'pd' (past day), 'pw' (past week),
                   'pm' (past month), 'py' (past year)
    """
    headers = {
        "Accept": "application/json",
        "Accept-Encoding": "gzip",
        "X-Subscription-Token": BRAVE_API_KEY,
    }
    params = {"q": query, "count": min(count, 20)}
    if freshness:
        params["freshness"] = freshness

    response = requests.get(BRAVE_SEARCH_URL, headers=headers,
                            params=params, timeout=30)
    response.raise_for_status()

    results = []
    data = response.json()
    for item in data.get("web", {}).get("results", []):
        results.append({
            "title": item["title"],
            "url": item["url"],
            "description": item.get("description", ""),
            "age": item.get("age", ""),
        })
    return results

Step 3: Extract content from sources

Fetch and extract readable content from the most relevant URLs:

from bs4 import BeautifulSoup

def extract_page_content(url):
    """Fetch a URL and extract the main text content."""
    headers = {"User-Agent": "research-bot/1.0.0"}
    response = requests.get(url, headers=headers, timeout=30)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")

    # Remove navigation, ads, footers
    for tag in soup.select("nav, footer, header, aside, .ad, .sidebar, script, style"):
        tag.decompose()

    # Try to find the main content area
    main = soup.select_one("article, main, .post-content, .entry-content")
    if not main:
        main = soup.find("body")

    if main:
        text = main.get_text(separator="\n", strip=True)
        # Clean up excessive whitespace
        lines = [line.strip() for line in text.split("\n") if line.strip()]
        return "\n".join(lines)
    return ""

Step 4: Analyze and cross-reference

For each sub-question, synthesize information from multiple sources:

Sub-question: "Rust web frameworks comparison"

Source 1 (blog.example.com): Actix-web is the fastest, Axum has the
  best developer experience, Rocket is easiest for beginners.

Source 2 (benchmark-site.com): Actix-web handles 650K req/s, Axum
  handles 580K req/s. Both outperform Express.js by 10x.

Source 3 (forum discussion): Community consensus is shifting toward
  Axum due to its use of the Tokio ecosystem and simpler API.

Synthesis: Axum is emerging as the recommended choice, offering a
  balance of performance and ergonomics, with strong ecosystem support.

Step 5: Compile the research report

# Research Report: [Topic]

**Date:** [date]
**Queries executed:** [count]
**Sources analyzed:** [count]

## Executive Summary

[2-3 paragraph summary of key findings]

## Detailed Findings

### [Sub-topic 1]

[Findings with inline citations]

Key data points:
- [fact] [Source 1]
- [fact] [Source 2]

### [Sub-topic 2]

[Findings with inline citations]

## Conflicting Information

[Note any areas where sources disagree and why]

## Knowledge Gaps

[Areas where information was insufficient or outdated]

## Sources

1. [Title](URL) - [brief description of what was extracted]
2. [Title](URL) - [brief description]
3. [Title](URL) - [brief description]
...

## Methodology

- Search engine: Brave Search API
- Queries executed: [list each query]
- Date range: [freshness filter used]
- Sources evaluated: [total count]
- Sources included: [count included in report]

Step 6: Save the report

cat > research_report_[topic].md << 'EOF'
[compiled report]
EOF

Examples

Example 1: Technology evaluation research

User request: "Research whether we should migrate from REST to GraphQL for our mobile app API."

Research plan:

1. "GraphQL vs REST mobile app performance"
2. "GraphQL migration challenges production"
3. "companies migrated REST to GraphQL results"
4. "GraphQL disadvantages drawbacks"
5. "REST vs GraphQL mobile bandwidth optimization"

Output: Structured report with pros, cons, case studies, and a recommendation based on evidence.

Example 2: Market research for a new product

User request: "Research the current state of AI code review tools. Who are the players and what are the gaps?"

Research plan:

1. "AI code review tools comparison 2025"
2. "AI code review market size growth"
3. "best AI code review GitHub" (product-specific)
4. "AI code review limitations complaints"
5. "developer survey code review automation"

Output: Competitive landscape analysis with feature matrices, pricing data, user sentiment, and identified market gaps.

Example 3: Fact-checking and verification

User request: "Research whether the claim 'microservices reduce deployment frequency' is supported by evidence."

Research plan:

1. "microservices deployment frequency study"
2. "microservices vs monolith deployment speed evidence"
3. "DORA metrics microservices research"
4. "microservices deployment challenges"

Output: Evidence-based analysis citing research papers, industry surveys, and case studies both supporting and refuting the claim.

Guidelines

• Always verify the Brave Search API key is set in the environment. Guide the user to get one at https://brave.com/search/api/ if missing (free tier: 2,000 queries/month).
• Execute multiple search queries per research topic. A single query is rarely sufficient for thorough research.
• Cross-reference claims across at least 2-3 sources before including them in the report.
• Clearly distinguish between facts, opinions, and the report's synthesis.
• Always include source URLs so the user can verify claims. Never present information without attribution.
• Note the publication date of sources. Older sources may contain outdated information.
• Flag conflicting information explicitly rather than silently choosing one version.
• Respect robots.txt and rate-limit page fetches. Add a 1-second delay between requests.
• If a page cannot be fetched (paywall, 403, timeout), note it and rely on the search snippet instead.
• Keep the final report focused and actionable. Raw information dumps are less useful than synthesized insights.
• Save the report in markdown format for easy reading and sharing.
• For time-sensitive topics, use the freshness parameter to prioritize recent results.