# Benchmarking System

> The Local Deep Research benchmarking system evaluates search configurations, models, and strategies using standardized datasets to help you optimize performance.

- Skill: `tools-only/benchmarking-system` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add tools-only/benchmarking-system`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/benchmarking-system/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-29
- Page: https://skillmd.com/skills/tools-only/benchmarking-system

---

# Benchmarking System

The Local Deep Research benchmarking system evaluates search configurations, models, and strategies using standardized datasets to help you optimize performance.

**Important**: Benchmark results are indicators for configuration testing, not predictors of performance on your specific research topics. What works well on SimpleQA may perform differently on your actual research questions.

## Quick Start

### Web Interface
1. Navigate to **Benchmark** in the web interface
2. Configure your test:
   - Select datasets (SimpleQA recommended)
   - Set number of examples (start with 20-50)
   - Uses your current Settings configuration
3. Click **Start Benchmark** and monitor progress
4. View results in **Benchmark Results** page

## Datasets

### SimpleQA (Recommended)
- Fact-based questions with clear answers
- Best for testing general knowledge retrieval
- Good baseline for comparing configurations

### BrowseComp (Advanced)
- Complex browsing and comparison tasks
- Currently limited performance - use max 20 examples for testing

## Configuration Options

### Search Engines
- **Tavily**: AI-optimized commercial API
- **SearXNG**: Meta-search aggregating multiple engines
- **Brave**: Independent search engine
- **Specialized engines** (ArXiv, PubMed, Wikipedia): Not suitable for general SimpleQA testing

### Strategies
- **Focused Iteration**: Best for SimpleQA fact-based questions
- **Source-Based**: Better for comprehensive research

## Interpreting Results

### Key Metrics
- **Accuracy**: Percentage of correct answers
- **Processing Time**: Time per question (30-60s is typical)
- **Search Results**: Number of search results retrieved per query

### Performance Expectations
- **Focused iteration with SimpleQA**: Around 95% potential with optimal setup
- **Source-based strategy**: Around 70% accuracy, more comprehensive results

## Best Practices

### Testing Workflow
1. Start with 20 examples to verify configuration
2. Check that search results are being retrieved
3. Scale to 50-100 examples for reliable metrics
4. Adjust settings based on results

### Troubleshooting
- **Low accuracy**: Verify API keys and search engine connectivity
- **No search results**: Check API credentials and rate limiting
- **Very fast processing**: Usually indicates configuration issues

## Requirements

### API Keys Needed
- **Evaluation**: OpenRouter API key for automatic grading
- **Search**: API key for your chosen search engine
- **LLM**: API key for your language model provider

## Responsible Usage

- Start with small tests to verify configuration
- Use moderate example counts for shared resources
- Monitor API usage in the Metrics page
- Respect rate limits and shared infrastructure

## Important Limitations

Benchmarks test standardized questions and may not reflect performance on:
- Your specific domain or research topics
- Complex, multi-step research questions
- Real-time or recent information queries
- Specialized knowledge areas

Use benchmarks as configuration guidance, then test with your actual research topics to validate performance.

---

The benchmarking system helps you find starting configurations for reliable research results.