Fs Researcher Scaling

FS-Researcher: Test-Time Scaling via External Memory

Problem

Research tasks require accumulating knowledge far exceeding model context windows. Deep research needs iterative refinement, synthesis across sources, and integration with external findings.

Single-pass generation cannot match quality of human iterative research. Standard scaling approaches hit context limits.

Core Concept

FS-Researcher uses the filesystem as durable external memory and coordination medium. A Context Builder agent accumulates knowledge by web browsing and extracting information; a Report Writer composes final output using the persistent knowledge base.

Allocating more computation rounds to the Context Builder directly improves downstream report quality, demonstrating effective test-time scaling.

Architecture Overview

• Context Builder Agent: Web browsing, information extraction, hierarchical knowledge base construction
• Report Writer Agent: Section-by-section composition using persistent knowledge base
• Filesystem Coordination: Shared external memory enabling cross-session persistence
• Hierarchical Storage: Organize findings into nested structures
• Iterative Refinement: Multiple rounds of context building improve knowledge base quality
• Section-by-Section Writing: Incremental report composition with knowledge base lookups

Implementation

Step 1: Build Knowledge Base Storage

Create hierarchical filesystem structure for knowledge accumulation.

import os
import json
from pathlib import Path

class KnowledgeBaseStorage:
    def __init__(self, base_path):
        self.base_path = Path(base_path)
        self.base_path.mkdir(parents=True, exist_ok=True)

    def add_finding(self, topic, finding_text, source_url, metadata=None):
        """Store a research finding."""
        topic_path = self.base_path / topic.replace(' ', '_')
        topic_path.mkdir(parents=True, exist_ok=True)

        # Create structured finding record
        finding = {
            'text': finding_text,
            'source': source_url,
            'timestamp': datetime.now().isoformat(),
            'metadata': metadata or {}
        }

        # Append to topic findings
        findings_file = topic_path / 'findings.jsonl'
        with open(findings_file, 'a') as f:
            f.write(json.dumps(finding) + '\n')

    def search_findings(self, topic, query):
        """Search findings within a topic."""
        topic_path = self.base_path / topic.replace(' ', '_')
        findings_file = topic_path / 'findings.jsonl'

        if not findings_file.exists():
            return []

        results = []
        with open(findings_file, 'r') as f:
            for line in f:
                finding = json.loads(line)
                if query.lower() in finding['text'].lower():
                    results.append(finding)

        return results

    def get_all_findings_for_topic(self, topic):
        """Retrieve all findings for a topic."""
        topic_path = self.base_path / topic.replace(' ', '_')
        findings_file = topic_path / 'findings.jsonl'

        if not findings_file.exists():
            return []

        findings = []
        with open(findings_file, 'r') as f:
            for line in f:
                findings.append(json.loads(line))

        return findings

Step 2: Context Builder Agent

Systematically accumulate knowledge through web search and extraction.

def context_builder_agent(research_query, kb_storage, search_engine, num_rounds=3):
    """Iteratively expand knowledge base for research topic."""
    for round_num in range(num_rounds):
        # Generate search queries based on current knowledge
        current_findings = kb_storage.get_all_findings_for_topic(research_query)

        if round_num == 0:
            # Initial searches
            search_queries = generate_initial_queries(research_query)
        else:
            # Refinement searches based on gaps
            gaps = identify_knowledge_gaps(current_findings)
            search_queries = generate_refinement_queries(gaps)

        # Search and extract
        for query in search_queries:
            results = search_engine.search(query)

            for result in results[:5]:  # Top 5 results
                # Extract key information
                extracted_text = extract_key_info(result['content'])

                # Store finding
                kb_storage.add_finding(
                    topic=research_query,
                    finding_text=extracted_text,
                    source_url=result['url'],
                    metadata={'round': round_num, 'query': query}
                )

    return kb_storage

Step 3: Report Writer Agent

Compose research report using knowledge base.

def report_writer_agent(research_query, kb_storage, report_model, num_sections=5):
    """Write comprehensive research report section-by-section."""
    # Generate section outlines
    all_findings = kb_storage.get_all_findings_for_topic(research_query)
    section_topics = generate_section_outline(research_query, len(all_findings))

    report = ""

    for section_idx, section_topic in enumerate(section_topics):
        # Find relevant findings for this section
        relevant_findings = kb_storage.search_findings(research_query, section_topic)

        # Synthesize findings into section context
        context = synthesize_findings(relevant_findings)

        # Generate section
        section_prompt = f"""Write a well-organized research section on: {section_topic}

Supporting findings:
{context}

Write a comprehensive, well-sourced section:"""

        section_text = report_model.generate(section_prompt, max_tokens=800)

        # Append section with citations
        report += f"\n## {section_topic}\n{section_text}\n"

    return report

Step 4: Test-Time Scaling Integration

Verify that additional context building rounds improve output quality.

def evaluate_scaling_correlation(research_query, kb_storage, report_model, max_rounds=5):
    """Validate positive correlation between computation and quality."""
    quality_scores = []

    for num_rounds in range(1, max_rounds + 1):
        # Build with varying rounds
        kb_storage = context_builder_agent(research_query, kb_storage, num_rounds=num_rounds)

        # Generate report
        report = report_writer_agent(research_query, kb_storage, report_model)

        # Evaluate quality
        score = evaluate_report_quality(report, research_query)
        quality_scores.append({
            'rounds': num_rounds,
            'quality_score': score
        })

    # Verify positive trend
    correlation = compute_correlation([s['rounds'] for s in quality_scores],
                                      [s['quality_score'] for s in quality_scores])

    return quality_scores, correlation > 0

Practical Guidance

Hyperparameter Configuration

When to Use

• Deep research tasks requiring comprehensive knowledge
• Multi-source synthesis (literature reviews, market research)
• Long-horizon fact-finding where quality improves iteratively
• Knowledge-intensive tasks exceeding context windows
• Systems with persistent storage available

When Not to Use

• Single-source lookups (web search sufficient)
• Real-time tasks requiring immediate responses
• Highly dynamic domains where information becomes stale
• Environments without persistent storage
• Tasks with clear, bounded information needs

Common Pitfalls

1. Knowledge base bloat: Unfiltered findings accumulate noise. Implement quality filters or deduplication.
2. Search query repetition: Multiple rounds may repeat searches. Track already-queried topics.
3. Synthesis hallucination: Report writer may invent citations. Require explicit source anchoring.
4. Scaling plateau: Improvement eventually plateaus. Validate diminishing returns and stop early.

Reference

FS-Researcher: Test-Time Scaling for Long-Horizon Research Tasks https://arxiv.org/abs/2602.01566