Search As Code

Search as Code (SaC)

An agentic search architecture where Claude generates Python code using a composable Search SDK, executes it in a sandbox, and persists intermediate state across turns via explicit filesystem serialization — producing higher-quality research with fewer tokens than serial function-call-based search.

How It Works

Rather than calling search as a black-box tool, Claude acts as the control plane: it writes a Python program that orchestrates the search stack, executes that program, inspects intermediate results, and iterates until it has enough evidence to synthesize.

The three layers:

1. Claude — reasons about strategy, generates code, decides when to synthesize
2. Sandbox — executes code deterministically (parallelism, filtering, aggregation)
3. Agentic Search SDK — composable primitives from raw retrieval to LLM subroutines

Variants

The CLI and browser variants share the same agent protocol but the browser version uses a richer response format — see Browser (JSX) variant below.

Setup

pip install anthropic rich requests
export ANTHROPIC_API_KEY=sk-...

Search backend — set one for real web search (otherwise results are simulated by Claude):

Without a search API key, SearchSDK falls back to asking Claude to generate plausible results. This is useful for demos and logic testing but not for real research — treat simulated results as structured placeholders only.

Sandbox hardening (production):

pip install llm-sandbox[mcp-docker]   # Docker-isolated execution

Set SANDBOX_BACKEND=docker to replace the default exec()-based sandbox with a fully isolated container. Recommended when the agent has network access or handles untrusted inputs.

Place sac.py anywhere on your path. Run:

python sac.py "Your research task here"
# or interactive:
python sac.py

SDK Reference

All primitives are available in generated code as sdk.*.

sdk.search

# Single web search
results = sdk.search.web(query, limit=8)          # -> list[SearchResult]

# Parallel web searches — queries can be strings or dicts with a 'query' key
results = sdk.search.web_many(queries, limit_per_query=8, concurrency=6)
# -> list[list[SearchResult]]

SearchResult fields: .url, .title, .snippet, .domain

Note: There is no sdk.search.neural() method and no mode= parameter on web_many in the base implementation. Neural/Exa search requires the extension described in Extending the SDK.

sdk.llm

# Synthesize/extract insights from a list of items
answer = sdk.llm.synthesize(items, instruction)       # -> str

# Generate a search plan
plan = sdk.llm.plan(context, goal)                    # -> str

# Batch-extract structured records matching a schema
# Items must be dicts. Internally chunks at 10 items per LLM call.
records = sdk.llm.extract_many(items, instruction, schema={
    "field_name": str,   # type hints guide extraction
    "matches": bool,
    "confidence": float,
})                                                     # -> list[dict]

sdk.fs — Filesystem state (persists across turns)

sdk.fs.write(key, data)       # serialize any Python object (pickle)
data = sdk.fs.read(key)       # deserialize
keys = sdk.fs.list()          # list persisted keys
exists = sdk.fs.exists(key)   # check before reading

Prefer sdk.fs over in-memory variables for anything needed in a later turn. Explicit serialization forces the model to be deliberate about what state matters.

sdk.utils

# ⚠ dedupe_by and filter_by require dict items, not SearchResult objects.
# Convert first: items = [vars(r) for r in results]  — or build dicts inline.
items = sdk.utils.dedupe_by(items, key="url")               # list[dict] -> list[dict]
items = sdk.utils.filter_by(items, field="domain", value="github.com")

summary = sdk.utils.summarize_coverage(items, by_fields=["vendor", "year"])
flat    = sdk.utils.flatten(list_of_lists)
text    = sdk.utils.join_result_fields(result)   # SearchResult -> "title | snippet"

No sdk.utils.chunk — do not call it; it will raise AttributeError. sdk.llm.extract_many already chunks at 10 items internally; pass the full list.

Agent Response Protocol

Each turn Claude returns one JSON object (no markdown fences):

Code turn — execute a search phase:

{
  "turn_type": "code",
  "reasoning": "Brief explanation of this phase",
  "code": "# Python using sdk\nresults = sdk.search.web_many([...])"
}

Synthesis turn — final answer:

{
  "turn_type": "synthesis",
  "reasoning": "What was found and how",
  "answer": "Complete answer to the task"
}

Search Strategy Patterns

Pattern 1 — Parallel fanout (Phase 1)

Fan out many query variants in parallel. Note: dedupe_by requires dicts — convert SearchResult objects before passing.

queries = [
    'site:arxiv.org "inference optimization" 2025',
    'site:github.com LLM serving throughput benchmark',
    '"speculative decoding" latency site:arxiv.org',
    '"KV cache compression" transformer 2024 2025',
]

raw = sdk.search.web_many(queries, limit_per_query=8, concurrency=6)
flat = sdk.utils.flatten(raw)

# Convert SearchResult -> dict before deduplication
items = [{"url": r.url, "title": r.title, "snippet": r.snippet, "domain": r.domain}
         for r in flat]
deduped = sdk.utils.dedupe_by(items, key="url")
sdk.fs.write("phase1_results", deduped)

Pattern 2 — Gap analysis + backfill (Phase 2)

After Phase 1, identify sparse areas and generate targeted follow-up queries.

results = sdk.fs.read("phase1_results")   # already dicts from Pattern 1
coverage = sdk.utils.summarize_coverage(results, by_fields=["domain"])
gap_queries = sdk.llm.synthesize(
    [{"coverage": coverage, "goal": "Find high-severity CVEs from 2024"}],
    instruction="Suggest 5 more site-scoped queries to fill coverage gaps. Return as JSON list."
)
# parse gap_queries and run sdk.search.web_many(...)

Pattern 3 — Structured extraction (Phase 3)

Extract typed records from raw results with confidence filtering. Pass dicts to extract_many; build them inline from SearchResult if needed.

raw = sdk.fs.read("phase1_results")   # already dicts
records = sdk.llm.extract_many(
    raw,   # extract_many chunks internally — pass the full list
    instruction="Extract papers where the abstract describes a new training method.",
    schema={
        "matches": bool,
        "title": str,
        "method_name": str,
        "key_claim": str,
        "source_url": str,
        "confidence": float,
    }
)
verified = [r for r in records if r.get("matches") and r.get("confidence", 0) > 0.75]
sdk.fs.write("verified_records", verified)

Pattern 4 — Inline LLM subroutine

Use sdk.llm.synthesize as an intermediate planning step mid-trajectory.

seed_results = sdk.fs.read("phase1_results")   # dicts
plan = sdk.llm.plan(
    context=str([r["title"] for r in seed_results[:20]]),
    goal="Identify which vendors have the most coverage gaps"
)
sdk.fs.write("gap_plan", plan)

Multi-Turn Trajectory Structure

A typical 3-turn trajectory:

For simple tasks (2 turns): fanout → synthesize. For complex tasks (4–5 turns): fanout → gap analysis → backfill → extract → synthesize.

Code Generation Guidelines

When writing code for the sandbox:

• One phase per turn — don't try to do everything in one code block
• Always persist to sdk.fs after collecting results — explicit serde outperforms REPL-style namespaces on long trajectories
• Use concurrency=6–12 for web_many to maximize parallel throughput
• Convert SearchResult to dict before utils calls — dedupe_by and filter_by call .get() and will raise AttributeError on dataclasses
• Don't pre-chunk for extract_many — it chunks at 10 internally; pre-chunking adds boilerplate for no gain
• Scope keyword queries precisely — site: prefixes, exact-phrase constraints ("CVE-2024"), year filters
• Filter early — use sdk.utils.filter_by and sdk.utils.dedupe_by before passing to LLM calls
• Confidence threshold — filter extract_many results by confidence > 0.75
• Check sdk.fs.list() at the start of turns 2+ to understand available state

Example Trajectories

CVE Advisory Research

# Turn 1: Fan out vendor-specific advisory queries
templates = [
    ("Mozilla", 'site:mozilla.org/security/advisories "CVE-2024" "Impact: high"'),
    ("Chrome",  'site:chromereleases.googleblog.com "High CVE-2024" stable'),
    ("Jenkins", 'site:jenkins.io/security/advisory "CVE-2024" "High" "Fix"'),
]
queries = [{"query": q} for _, q in templates]
hits = sdk.search.web_many(queries, limit_per_query=8, concurrency=12)
pages = sdk.utils.dedupe_by(
    [{"url": r.url, "title": r.title, "snippet": r.snippet, "domain": r.domain}
     for r in sdk.utils.flatten(hits)],
    key="url"
)
sdk.fs.write("advisory_pages", pages)

# Turn 2: Extract structured CVE records
pages = sdk.fs.read("advisory_pages")   # dicts
records = sdk.llm.extract_many(pages,
    instruction="Extract CVEs where a vendor advisory names the fixed version.",
    schema={"cve": str, "vendor": str, "product": str,
            "fix_version": str, "source_url": str,
            "version_bound_to_cve": bool, "confidence": float}
)
verified = [r for r in records if r.get("version_bound_to_cve") and r.get("confidence", 0) > 0.75]
sdk.fs.write("cve_records", verified)

Competitive Analysis

# Turn 1: Parallel per-company searches
companies = ["OpenAI", "Anthropic", "Google DeepMind", "Mistral", "Meta AI"]
queries = [f"{co} model release announcement 2025" for co in companies]
raw = sdk.search.web_many(queries, limit_per_query=6, concurrency=8)
results = [{"url": r.url, "title": r.title, "snippet": r.snippet, "domain": r.domain,
            "company": co}
           for co, batch in zip(companies, raw) for r in batch]
sdk.fs.write("company_results", results)

Browser (JSX) Variant

search-as-code.jsx is a self-contained React artifact that calls the Anthropic API directly from the browser. It uses a richer response format and simulates search via a second Claude call (no real search backend by default).

Extended response format (browser only):

Code turn:

{
  "turn_type": "code",
  "reasoning": "...",
  "code": "# Python-style pseudocode shown in UI",
  "operations": [
    {"type": "web_search", "queries": ["query1", "query2"]},
    {"type": "fs_write",   "key": "phase1_results"},
    {"type": "fs_read",    "key": "phase1_results"},
    {"type": "llm_call",   "purpose": "gap analysis"}
  ],
  "next_turn_hint": "What the next turn will do"
}

Synthesis turn:

{
  "turn_type": "synthesis",
  "reasoning": "...",
  "answer": "Complete answer",
  "sources": ["url1", "url2"],
  "stats": {
    "searches_run": 12,
    "results_processed": 87,
    "token_estimate": "~42K"
  }
}

The operations array drives actual execution: web_search operations trigger executeSearches(queries), and fs_write/fs_read operations update the simulated currentFs dict in React state. The Python code field is rendered for display only — it is not executed in the browser.

Extending the SDK

The AgenticSearchSDK class in sac.py has four sub-SDKs you can extend:

• SearchSDK — add retrieval backends; _search_one(query, limit) is the extension point; must return list[SearchResult]
• LLMSDKClient — add extraction/synthesis patterns
• FilesystemSDK — swap pickle for JSON, SQLite, or a real key-value store
• UtilsSDK — add domain-specific helpers

Wiring Exa (neural search) — subclass SearchSDK:

from exa_py import Exa

class ExaSearchSDK(SearchSDK):
    def __init__(self, client, brave_key=None):
        super().__init__(client, brave_key)
        self._exa = Exa(api_key=os.environ["EXA_API_KEY"])

    def neural(self, query: str, limit: int = 8) -> list[SearchResult]:
        """Semantic / embedding-based search via Exa."""
        results = self._exa.search_and_contents(query, num_results=limit, use_autoprompt=True)
        return [SearchResult(url=r.url, title=r.title or "", snippet=(r.text or "")[:300])
                for r in results.results]

# Instantiate with ExaSearchSDK instead of the default:
sdk = AgenticSearchSDK.__new__(AgenticSearchSDK)
sdk.search = ExaSearchSDK(client, brave_key)
sdk.llm    = LLMSDKClient(client)
sdk.fs     = FilesystemSDK(fs_dir)
sdk.utils  = UtilsSDK()

Once wired, call sdk.search.neural(query, limit) from generated code.

Wiring llm-sandbox (Docker isolation):

# Replace Sandbox.execute() with:
from llm_sandbox import SandboxSession
with SandboxSession(lang="python", keep_template=True) as session:
    result = session.run(code)

Set SANDBOX_BACKEND=docker to activate. Provides CPU/memory limits, network isolation, and automatic cleanup — essential when the agent accesses the open web.

Troubleshooting

AttributeError: 'SearchResult' object has no attribute 'get' — you passed list[SearchResult] to dedupe_by or filter_by. Convert first: items = [{"url": r.url, "title": r.title, "snippet": r.snippet, "domain": r.domain} for r in results]

AttributeError: 'UtilsSDK' object has no attribute 'chunk' — chunk is not implemented. Remove the call; extract_many chunks at 10 items internally.

AttributeError: 'SearchSDK' object has no attribute 'neural' — neural() is not in the base SearchSDK. Use the Exa extension above or fall back to sdk.search.web().

JSON parse errors from agent — the model returned markdown-wrapped JSON; the runtime strips fences automatically, but very long outputs can overflow max_tokens. Increase it in SaCAgent.

Sandbox execution errors — printed with a traceback; the agent receives the error and can self-correct on the next turn. If errors recur, check that the generated code only references SDK methods that exist.

Empty or low-quality results — without a real search backend, results are simulated by Claude. For production tasks set BRAVE_SEARCH_API_KEY or wire Exa. Simulated results are useful for validating pipeline logic but should not be trusted for factual research.

Too many turns — default max_turns=6. Increase in SaCAgent.__init__ for complex tasks (e.g. 200+ record extraction).

Sandbox security — the default exec()-based sandbox has no isolation. For tasks involving untrusted input or open-web access, use llm-sandbox with Docker backend.

Evals

Test cases live in evals/evals.json. Run against a task to verify the skill triggers correctly and produces structured, accurate output.

{
  "skill_name": "search-as-code",
  "evals": [
    {
      "id": 1,
      "prompt": "Find the five most-cited papers on speculative decoding published in 2024, with author names and arxiv links.",
      "expected_output": "A list of 5 papers with titles, authors, and arxiv URLs, sourced from real search results across multiple turns."
    },
    {
      "id": 2,
      "prompt": "Compare the pricing and context window of the latest models from OpenAI, Anthropic, and Google as of today.",
      "expected_output": "A structured comparison table with accurate, sourced data — not from training knowledge."
    },
    {
      "id": 3,
      "prompt": "List all high-severity CVEs patched by Mozilla in 2024 with fix versions from official advisories.",
      "expected_output": "Structured records with CVE IDs, products, fix versions, and mozilla.org source URLs. Confidence > 0.75 on all records."
    }
  ]
}