tools-only Benchmarking System
The Local Deep Research benchmarking system evaluates search configurations, models, and strategies using standardized datasets to help you optimize performance.
Important: Benchmark results are indicators for configuration testing, not predictors of performance on your specific research topics. What works well on SimpleQA may perform differently on your actual research questions.
Quick Start
Web Interface
- Navigate to Benchmark in the web interface
- Configure your test:
- Select datasets (SimpleQA recommended)
- Set number of examples (start with 20-50)
- Uses your current Settings configuration
- Click Start Benchmark and monitor progress
- View results in Benchmark Results page
Datasets
SimpleQA (Recommended)
- Fact-based questions with clear answers
- Best for testing general knowledge retrieval
- Good baseline for comparing configurations
BrowseComp (Advanced)
- Complex browsing and comparison tasks
- Currently limited performance - use max 20 examples for testing
Configuration Options
Search Engines
- Tavily: AI-optimized commercial API
- SearXNG: Meta-search aggregating multiple engines
- Brave: Independent search engine
- Specialized engines (ArXiv, PubMed, Wikipedia): Not suitable for general SimpleQA testing
Strategies
- Focused Iteration: Best for SimpleQA fact-based questions
- Source-Based: Better for comprehensive research
Interpreting Results
Key Metrics
- Accuracy: Percentage of correct answers
- Processing Time: Time per question (30-60s is typical)
- Search Results: Number of search results retrieved per query
Performance Expectations
- Focused iteration with SimpleQA: Around 95% potential with optimal setup
- Source-based strategy: Around 70% accuracy, more comprehensive results
Best Practices
Testing Workflow
- Start with 20 examples to verify configuration
- Check that search results are being retrieved
- Scale to 50-100 examples for reliable metrics
- Adjust settings based on results
Troubleshooting
- Low accuracy: Verify API keys and search engine connectivity
- No search results: Check API credentials and rate limiting
- Very fast processing: Usually indicates configuration issues
Requirements
API Keys Needed
- Evaluation: OpenRouter API key for automatic grading
- Search: API key for your chosen search engine
- LLM: API key for your language model provider
Responsible Usage
- Start with small tests to verify configuration
- Use moderate example counts for shared resources
- Monitor API usage in the Metrics page
- Respect rate limits and shared infrastructure
Important Limitations
Benchmarks test standardized questions and may not reflect performance on:
- Your specific domain or research topics
- Complex, multi-step research questions
- Real-time or recent information queries
- Specialized knowledge areas
Use benchmarks as configuration guidance, then test with your actual research topics to validate performance.
The benchmarking system helps you find starting configurations for reliable research results.