Search Engines Guide
Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.
Note: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via GitHub Issues.
Overview
LDR supports three categories of search engines:
- Free Search Engines - No API key required
- Premium Search Engines - Require API keys but offer enhanced features
- Custom Sources - Your own documents and databases
Search Engine Selection
Auto Engine (Recommended)
The auto search engine intelligently selects the most appropriate engines based on your query:
result = quick_summary(
query="What are the latest advances in quantum computing?",
search_tool="auto" # Automatically selects best engines
)
Free Search Engines
Academic Search Engines
arXiv
- Specialization: Scientific papers and preprints
- Best for: Physics, mathematics, computer science, biology
- Results: Direct access to research papers
- Rate Limit: Moderate - automatic retry on limits
PubMed
- Specialization: Biomedical and life science literature
- Best for: Medical research, clinical studies, biology
- Results: Abstracts and links to full papers
- Rate Limit: Generous - rarely hits limits
Semantic Scholar
- Specialization: Academic literature across all fields
- Best for: Cross-disciplinary research, citation networks
- Results: Paper summaries with citation context
- Rate Limit: Moderate - adaptive rate limiting handles this
General Purpose
Wikipedia
- Specialization: General knowledge and encyclopedic information
- Best for: Background information, concepts, facts
- Results: Well-structured article content
- Rate Limit: Very generous
SearXNG (Highly Recommended)
DuckDuckGo
- Specialization: Privacy-focused web search
- Best for: General web queries without tracking
- Results: Web pages, instant answers
- Rate Limit: Strict - use SearXNG for better reliability
Technical Search
GitHub
- Specialization: Code repositories and documentation
- Best for: Finding code examples, libraries, technical solutions
- Results: Repository information, code snippets, issues
- Rate Limit: Moderate when unauthenticated
Elasticsearch
- Specialization: Custom search within your Elasticsearch cluster
- Best for: Searching your own indexed data
- Configuration: See Elasticsearch Setup Guide
Historical Search
Wayback Machine
- Specialization: Historical web content
- Best for: Finding deleted content, tracking changes over time
- Results: Archived web pages with timestamps
- Rate Limit: Moderate
News Search
The Guardian
- Specialization: News articles and journalism
- Best for: Current events, news analysis
- Results: Recent news articles
- Note: Requires API key (free tier available at https://open-platform.theguardian.com/)
Wikinews
- Specialization: Open and collaboratively-written news articles on a wide range of topics
- Best for: Historical and recent news, general news coverage, quick overviews
- Results: News articles written by volunteers with verified sources
Premium Search Engines
Tavily
Google (via SerpAPI)
Google Programmable Search Engine
Brave Search
Custom Sources
Local Documents
- Specialization: Search your private documents
- Supported formats: PDF, TXT, MD, DOCX, CSV, and more
- Configuration: See Configuring Local Search
- Setup:
- Go to Settings → Search for "local"
- Add document collection paths
- Choose embedding model (CPU or Ollama)
- First search will index documents
LangChain Retrievers
- Specialization: Any vector store or database
- Supported: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch
- Configuration: See LangChain Integration Guide
Search Performance Comparison
| Engine |
Speed |
Quality |
Privacy |
Rate Limits |
| SearXNG |
★★★★★ |
★★★★☆ |
★★★★★ |
★★★★★ |
| Wikipedia |
★★★★☆ |
★★★★☆ |
★★★★★ |
★★★★★ |
| arXiv |
★★★★☆ |
★★★★★ |
★★★★★ |
★★★☆☆ |
| PubMed |
★★★★☆ |
★★★★★ |
★★★★★ |
★★★★☆ |
| Tavily |
★★★★☆ |
★★★★★ |
★★★☆☆ |
★★★★☆ |
| Google (SerpAPI) |
★★★★☆ |
★★★★★ |
★★☆☆☆ |
★★★★★ |
| Local Documents |
★★★☆☆ |
★★★★★ |
★★★★★ |
★★★★★ |
Rate Limiting and Reliability
LDR includes intelligent adaptive rate limiting that:
- Learns optimal wait times for each engine
- Automatically retries failed requests
- Prevents your IP from being blocked
- Maintains high reliability
Managing Rate Limits
# Check rate limit status
python -m local_deep_research.web_search_engines.rate_limiting status
# Reset rate limits if needed
python -m local_deep_research.web_search_engines.rate_limiting reset
Search Strategies
LDR supports multiple search strategies that determine how queries are processed:
- source-based: Single query, fast results
- focused_iteration: Iterative refinement for accuracy (recommended)
Best Practices
- For General Research: Use
auto or searxng
- For Academic Research: Combine
arxiv, pubmed, and semantic_scholar
- For Technical Questions: Use
github with searxng
- For Maximum Privacy: Use
searxng with local Ollama models
- For Best Quality: Use
tavily or Google with focused_iteration strategy
Troubleshooting
SearXNG Not Working
- Verify container is running:
docker ps | grep searxng
- Check URL in settings:
http://localhost:8080
- Test directly:
curl http://localhost:8080
- Check the logs:
docker logs searxng or view them in the LDR web UI
Rate Limit Errors
- Wait a few minutes and retry
- Use
auto engine for automatic fallbacks
- Consider adding premium engines for higher limits
No Results Found
- Try different search engines
- Broaden your query
- Check internet connectivity
- Verify API keys for premium engines
Advanced Configuration
Configuring Search Engines
You can enable/disable specific search engines and adjust their reliability parameters in the settings. This affects which engines are used in 'auto' mode and how the system handles rate limiting.
Parallel Search Configuration
Adjust parallel search settings:
result = quick_summary(
query="your query",
max_parallel_searches=5, # Number of engines to query simultaneously
timeout_per_engine=30 # Seconds per engine
)
Related Documentation
- API Quickstart
- Configuration Guide
- LangChain Integration
- Elasticsearch Setup
1---2name: search-engines-guide3description: Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.4---5# Search Engines Guide67Local Deep Research integrates with multiple search engines to provide comprehensive research capabilities. This guide covers all available search engines, their specializations, and configuration details.89> **Note**: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues).1011## Overview1213LDR supports three categories of search engines:14- **Free Search Engines** - No API key required15- **Premium Search Engines** - Require API keys but offer enhanced features16- **Custom Sources** - Your own documents and databases1718## Search Engine Selection1920### Auto Engine (Recommended)2122The `auto` search engine intelligently selects the most appropriate engines based on your query:2324```python25result = quick_summary(26 query="What are the latest advances in quantum computing?",27 search_tool="auto" # Automatically selects best engines28)29```3031## Free Search Engines3233### Academic Search Engines3435#### arXiv36- **Specialization**: Scientific papers and preprints37- **Best for**: Physics, mathematics, computer science, biology38- **Results**: Direct access to research papers39- **Rate Limit**: Moderate - automatic retry on limits4041#### PubMed42- **Specialization**: Biomedical and life science literature43- **Best for**: Medical research, clinical studies, biology44- **Results**: Abstracts and links to full papers45- **Rate Limit**: Generous - rarely hits limits4647#### Semantic Scholar48- **Specialization**: Academic literature across all fields49- **Best for**: Cross-disciplinary research, citation networks50- **Results**: Paper summaries with citation context51- **Rate Limit**: Moderate - adaptive rate limiting handles this5253### General Purpose5455#### Wikipedia56- **Specialization**: General knowledge and encyclopedic information57- **Best for**: Background information, concepts, facts58- **Results**: Well-structured article content59- **Rate Limit**: Very generous6061#### SearXNG (Highly Recommended)62- **Specialization**: Meta-search engine aggregating multiple sources63- **Best for**: Comprehensive web search with privacy64- **Results**: Aggregated results from Google, Bing, DuckDuckGo, etc.65- **Setup**:66 ```bash67 docker pull searxng/searxng68 docker run -d -p 8080:8080 --name searxng searxng/searxng69 ```70- **Configuration**: Set URL to `http://localhost:8080` in settings7172#### DuckDuckGo73- **Specialization**: Privacy-focused web search74- **Best for**: General web queries without tracking75- **Results**: Web pages, instant answers76- **Rate Limit**: Strict - use SearXNG for better reliability7778### Technical Search7980#### GitHub81- **Specialization**: Code repositories and documentation82- **Best for**: Finding code examples, libraries, technical solutions83- **Results**: Repository information, code snippets, issues84- **Rate Limit**: Moderate when unauthenticated8586#### Elasticsearch87- **Specialization**: Custom search within your Elasticsearch cluster88- **Best for**: Searching your own indexed data89- **Configuration**: See [Elasticsearch Setup Guide](elasticsearch_search_engine.md)9091### Historical Search9293#### Wayback Machine94- **Specialization**: Historical web content95- **Best for**: Finding deleted content, tracking changes over time96- **Results**: Archived web pages with timestamps97- **Rate Limit**: Moderate9899### News Search100101#### The Guardian102- **Specialization**: News articles and journalism103- **Best for**: Current events, news analysis104- **Results**: Recent news articles105- **Note**: Requires API key (free tier available at https://open-platform.theguardian.com/)106107#### Wikinews108- **Specialization**: Open and collaboratively-written news articles on a wide range of topics109- **Best for**: Historical and recent news, general news coverage, quick overviews110- **Results**: News articles written by volunteers with verified sources111112## Premium Search Engines113114### Tavily115- **Specialization**: AI-optimized search for LLM applications116- **Best for**: High-quality, relevant results for AI research117- **Pricing**: Free tier available, paid plans for higher volume118- **Configuration**:119 ```bash120 # In .env file or web interface121 LDR_SEARCH_ENGINE_TAVILY_API_KEY=your-key-here122 ```123124### Google (via SerpAPI)125- **Specialization**: Comprehensive web search126- **Best for**: Most current and comprehensive results127- **Pricing**: Paid service with free trial128- **Configuration**:129 ```bash130 LDR_SEARCH_ENGINE_WEB_SERPAPI_API_KEY=your-key-here131 ```132133### Google Programmable Search Engine134- **Specialization**: Customizable Google search135- **Best for**: Searching specific sites or topics136- **Pricing**: Free tier with limits137- **Configuration**:138 ```bash139 LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_API_KEY=your-key-here140 LDR_SEARCH_ENGINE_WEB_GOOGLE_PSE_ENGINE_ID=your-engine-id141 ```142143### Brave Search144- **Specialization**: Independent search index with privacy focus145- **Best for**: Web search without big tech tracking146- **Pricing**: Free tier available147- **Configuration**:148 ```bash149 LDR_SEARCH_ENGINE_WEB_BRAVE_API_KEY=your-key-here150 ```151152## Custom Sources153154### Local Documents155- **Specialization**: Search your private documents156- **Supported formats**: PDF, TXT, MD, DOCX, CSV, and more157- **Configuration**: See [Configuring Local Search](https://github.com/LearningCircuit/local-deep-research/wiki/Configuring-Local-Search)158- **Setup**:159 1. Go to Settings → Search for "local"160 2. Add document collection paths161 3. Choose embedding model (CPU or Ollama)162 4. First search will index documents163164### LangChain Retrievers165- **Specialization**: Any vector store or database166- **Supported**: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch167- **Configuration**: See [LangChain Integration Guide](LANGCHAIN_RETRIEVER_INTEGRATION.md)168169170## Search Performance Comparison171172| Engine | Speed | Quality | Privacy | Rate Limits |173|--------|-------|---------|---------|-------------|174| SearXNG | ★★★★★ | ★★★★☆ | ★★★★★ | ★★★★★ |175| Wikipedia | ★★★★☆ | ★★★★☆ | ★★★★★ | ★★★★★ |176| arXiv | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★☆☆ |177| PubMed | ★★★★☆ | ★★★★★ | ★★★★★ | ★★★★☆ |178| Tavily | ★★★★☆ | ★★★★★ | ★★★☆☆ | ★★★★☆ |179| Google (SerpAPI) | ★★★★☆ | ★★★★★ | ★★☆☆☆ | ★★★★★ |180| Local Documents | ★★★☆☆ | ★★★★★ | ★★★★★ | ★★★★★ |181182## Rate Limiting and Reliability183184LDR includes intelligent adaptive rate limiting that:185- Learns optimal wait times for each engine186- Automatically retries failed requests187- Prevents your IP from being blocked188- Maintains high reliability189190### Managing Rate Limits191192```bash193# Check rate limit status194python -m local_deep_research.web_search_engines.rate_limiting status195196# Reset rate limits if needed197python -m local_deep_research.web_search_engines.rate_limiting reset198```199200## Search Strategies201202LDR supports multiple search strategies that determine how queries are processed:203204- **source-based**: Single query, fast results205- **focused_iteration**: Iterative refinement for accuracy (recommended)206207## Best Practices2082091. **For General Research**: Use `auto` or `searxng`2102. **For Academic Research**: Combine `arxiv`, `pubmed`, and `semantic_scholar`2113. **For Technical Questions**: Use `github` with `searxng`2124. **For Maximum Privacy**: Use `searxng` with local Ollama models2135. **For Best Quality**: Use `tavily` or Google with `focused_iteration` strategy214215## Troubleshooting216217### SearXNG Not Working218- Verify container is running: `docker ps | grep searxng`219- Check URL in settings: `http://localhost:8080`220- Test directly: `curl http://localhost:8080`221- Check the logs: `docker logs searxng` or view them in the LDR web UI222223### Rate Limit Errors224- Wait a few minutes and retry225- Use `auto` engine for automatic fallbacks226- Consider adding premium engines for higher limits227228### No Results Found229- Try different search engines230- Broaden your query231- Check internet connectivity232- Verify API keys for premium engines233234## Advanced Configuration235236### Configuring Search Engines237238You can enable/disable specific search engines and adjust their reliability parameters in the settings. This affects which engines are used in 'auto' mode and how the system handles rate limiting.239240### Parallel Search Configuration241242Adjust parallel search settings:243```python244result = quick_summary(245 query="your query",246 max_parallel_searches=5, # Number of engines to query simultaneously247 timeout_per_engine=30 # Seconds per engine248)249```250251## Related Documentation252253- [API Quickstart](api-quickstart.md)254- [Configuration Guide](env_configuration.md)255- [LangChain Integration](LANGCHAIN_RETRIEVER_INTEGRATION.md)256- [Elasticsearch Setup](elasticsearch_search_engine.md)
tools-only