Over Searching Control

Problem

Search-augmented LLMs exhibit a critical but overlooked inefficiency:

1. Over-Searching: Models invoke search tools unnecessarily for questions they could answer from training knowledge
2. Context Pollution: Irrelevant search results introduce noise that degrades reasoning
3. Contradictory Evidence: Retrieved documents may contradict model knowledge, confusing final answers
4. Abstention Failure: Models particularly struggle to abstain (say "I don't know") when search returns irrelevant documents
5. Token Waste: Unnecessary search rounds consume tokens that could improve other aspects of performance

Current systems treat search as always beneficial, but indiscriminate search actually harms performance on unanswerable questions.

Solution

Over-Searching Analysis introduces three components:

1. Tokens Per Correctness (TPC): A cost-aware metric specifically designed for search-augmented systems
   • Measures: How many tokens required per correctly answered question?
   • Accounts for both search cost and reasoning quality
2. Selective Search Triggering: Train models to recognize question types:
   • Answerable from training knowledge → skip search
   • Requires Current Information → search needed
   • Factual Lookup → search optimal
   • Unanswerable → don't search (prevents hallucination)
3. Evidence Quality Management: Incorporate negative evidence (counter-signals) to teach models when to ignore irrelevant search results

When to Use

• Token-Constrained Systems: Production systems where search cost matters
• Hybrid QA Agents: Systems combining training knowledge with retrieval
• Retrieval-Augmented Generation: Any RAG pipeline seeking efficiency gains
• Low-Confidence Detection: Agents that must recognize and abstain on unanswerable queries
• Multi-Turn Dialogue: Long conversations where accumulated search costs compound

When NOT to Use

• For pure retrieval systems (search is the only knowledge source)
• When knowledge freshness is critical (always search for current info)
• In systems where computational cost is unconstrained
• For specialized domains with no useful training knowledge

Core Concepts

The framework operates on the principle that search is a tool, not a default:

1. Selective Invocation: Use search strategically, not habitually
2. Cost-Aware Evaluation: Judge search value by tokens spent, not just accuracy
3. Confidence Gating: Only search when model confidence is low
4. Evidence Filtering: Train to recognize and ignore irrelevant retrieval results

Key Implementation Pattern

Building selective search behavior:

# Conceptual: selective search with confidence gating
class SelectiveSearchLLM:
    def answer_query(self, query):
        # Step 1: Attempt answer without search
        answer, confidence = self.generate_answer(query)

        # Step 2: Confidence-gated search
        if confidence < search_threshold:
            # Search only if confidence low
            search_results = self.retrieve(query)

            # Step 3: Incorporate evidence with filtering
            if self.is_relevant(search_results, answer):
                answer = self.refine_with_search(
                    query, answer, search_results
                )
            # else: ignore irrelevant results, keep original answer
        else:
            # High confidence: skip search entirely
            pass

        return answer

Key mechanisms:

• Confidence scoring before search invocation
• Evidence relevance filtering (ignore contradictory/irrelevant results)
• Negative example training (learning when NOT to search)
• Cost-benefit analysis (search tokens vs. accuracy improvement)

Expected Outcomes

• 20-30% Token Reduction: Fewer unnecessary search invocations
• Improved Accuracy: Avoiding irrelevant search results actually improves performance
• Better Abstention: Models correctly say "I don't know" on unanswerable questions
• Cost Optimization: Similar answer quality with significantly lower token budget

Limitations and Considerations

• Requires training data labeling question answerability (OverSearchQA dataset helps)
• Mismatch between training knowledge and fine-tuning data can create uncertainty
• Non-stationary information (time-sensitive facts) still requires search even if confidence is high
• Confidence calibration is model-dependent and may require tuning

Integration Pattern

For a production QA system:

1. Receive Query: "What is the capital of France?"
2. Estimate Confidence: High confidence (training knowledge)
3. Skip Search: Omit retrieval step, saving tokens
4. Return Answer: Paris

Versus:

1. Receive Query: "What are today's top news events?"
2. Estimate Confidence: Low (requires current information)
3. Search: Retrieve today's news
4. Synthesize Answer: Incorporate latest information

This selective approach optimizes both accuracy and efficiency.

OverSearchQA Dataset

The released OverSearchQA dataset provides 18,000+ QA examples labeled by answerability:

• Answerable from training knowledge
• Requires recent information
• Factual but searchable
• Unanswerable (trick questions, contradictions)

Use for training selective search models.

Related Work Context

Over-Searching Analysis challenges the assumption that search improves all QA tasks. By recognizing that search can harm performance on unanswerable questions, it enables more intelligent resource allocation in RAG systems.