Web Search API Integration
Pick the right provider, wire auth the way each API actually expects, and don't burn quota.
Provider selection
| Situation |
Use |
| Default LLM-friendly search |
Tavily |
| Conceptual / semantic queries |
Exa |
| Need page content (not snippets) |
Firecrawl |
| Freshness-critical lookups |
Brave Search API |
| Exact Google SERP / answer boxes |
Serper.dev or Google CSE |
| Unlimited dev/testing |
Self-hosted SearXNG |
Free tiers (verified 2026-08-24): Tavily 1,000 credits/mo · Exa $20 signup + $10/mo ·
Firecrawl 1,000 credits/mo · Brave $5/mo credits (card required since Aug 2025) ·
Serper 2,500 one-time · Google CSE 100 queries/day · SearXNG unlimited self-hosted.
Auth shapes (the part everyone gets wrong)
| Provider |
Auth location |
Env var convention |
| Tavily |
api_key in JSON body |
TAVILY_API_KEY |
| Exa |
x-api-key header |
EXA_API_KEY |
| Firecrawl |
Authorization: Bearer header |
FIRECRAWL_API_KEY |
| Brave |
X-Subscription-Token header |
BRAVE_SEARCH_API_KEY (never BRAVE_API_KEY) |
| Serper |
X-API-KEY header |
SERPER_API_KEY |
| Google CSE |
key + cx query params |
GCS_API_KEY + GCS_CX |
| SearXNG |
none (JSON format must be enabled server-side) |
n/a |
Reference implementations
import os, requests
def tavily_search(q, n=5):
r = requests.post("https://api.tavily.com/search", json={
"api_key": os.environ["TAVILY_API_KEY"], "query": q,
"search_depth": "basic", "max_results": n}, timeout=30)
r.raise_for_status()
return [{"title": x["title"], "url": x["url"], "snippet": x["content"]}
for x in r.json()["results"]]
def exa_search(q, n=5):
r = requests.post("https://api.exa.ai/search",
headers={"x-api-key": os.environ["EXA_API_KEY"]},
json={"query": q, "numResults": n}, timeout=30)
r.raise_for_status()
return [{"title": x["title"], "url": x["url"]} for x in r.json()["results"]]
def brave_search(q, n=5):
r = requests.get("https://api.search.brave.com/res/v1/web/search",
headers={"X-Subscription-Token": os.environ["BRAVE_SEARCH_API_KEY"],
"Accept": "application/json"},
params={"q": q, "count": n}, timeout=30)
r.raise_for_status()
return [{"title": x["title"], "url": x["url"], "snippet": x["description"],
"age": x.get("age")} for x in r.json()["web"]["results"]]
def serper_search(q, n=5):
r = requests.post("https://google.serper.dev/search",
headers={"X-API-KEY": os.environ["SERPER_API_KEY"]},
json={"q": q, "num": n}, timeout=20)
r.raise_for_status()
return [{"title": x["title"], "url": x["link"], "snippet": x["snippet"]}
for x in r.json().get("organic", [])[:n]]
Quota discipline
- Only HTTP 200 counts against real quota — log 429/5xx separately, never increment.
- Cache by normalized URL; identical repeat queries should never re-hit the API.
- Route cheap dev traffic to SearXNG; save metered keys for production paths.
- Track monthly usage per key in a state file; alert at 80% before exhaustion.
- Monthly refills reset on the 1st and do not roll over.
Pitfalls
- Tavily: key in body, not header.
advanced depth = 2× credits.
- Exa: no relevance scores returned;
/contents costs extra beyond /search.
- Firecrawl: crawl jobs are async (job ID first, poll later) — not empty results.
- Brave: card required at signup now; summarizer/grounding endpoints bill faster than plain search.
- Google CSE: needs BOTH
key and cx; quota resets midnight PT not IST.
1---2name: web-search-api3description: Use when an agent needs live web results — pick a search provider (Tavily, Exa, Firecrawl, Brave, Serper, Google CSE, SearXNG), authenticate, call it correctly, and stay inside free-tier quotas.4---56# Web Search API Integration78Pick the right provider, wire auth the way each API actually expects, and don't burn quota.910## Provider selection1112| Situation | Use |13|---|---|14| Default LLM-friendly search | Tavily |15| Conceptual / semantic queries | Exa |16| Need page content (not snippets) | Firecrawl |17| Freshness-critical lookups | Brave Search API |18| Exact Google SERP / answer boxes | Serper.dev or Google CSE |19| Unlimited dev/testing | Self-hosted SearXNG |2021Free tiers (verified 2026-08-24): Tavily 1,000 credits/mo · Exa $20 signup + $10/mo ·22Firecrawl 1,000 credits/mo · Brave $5/mo credits (card required since Aug 2025) ·23Serper 2,500 one-time · Google CSE 100 queries/day · SearXNG unlimited self-hosted.2425## Auth shapes (the part everyone gets wrong)2627| Provider | Auth location | Env var convention |28|---|---|---|29| Tavily | `api_key` **in JSON body** | `TAVILY_API_KEY` |30| Exa | `x-api-key` header | `EXA_API_KEY` |31| Firecrawl | `Authorization: Bearer` header | `FIRECRAWL_API_KEY` |32| Brave | `X-Subscription-Token` header | `BRAVE_SEARCH_API_KEY` (never `BRAVE_API_KEY`) |33| Serper | `X-API-KEY` header | `SERPER_API_KEY` |34| Google CSE | `key` + `cx` query params | `GCS_API_KEY` + `GCS_CX` |35| SearXNG | none (JSON format must be enabled server-side) | n/a |3637## Reference implementations3839```python40import os, requests4142def tavily_search(q, n=5):43 r = requests.post("https://api.tavily.com/search", json={44 "api_key": os.environ["TAVILY_API_KEY"], "query": q,45 "search_depth": "basic", "max_results": n}, timeout=30)46 r.raise_for_status()47 return [{"title": x["title"], "url": x["url"], "snippet": x["content"]}48 for x in r.json()["results"]]4950def exa_search(q, n=5):51 r = requests.post("https://api.exa.ai/search",52 headers={"x-api-key": os.environ["EXA_API_KEY"]},53 json={"query": q, "numResults": n}, timeout=30)54 r.raise_for_status()55 return [{"title": x["title"], "url": x["url"]} for x in r.json()["results"]]5657def brave_search(q, n=5):58 r = requests.get("https://api.search.brave.com/res/v1/web/search",59 headers={"X-Subscription-Token": os.environ["BRAVE_SEARCH_API_KEY"],60 "Accept": "application/json"},61 params={"q": q, "count": n}, timeout=30)62 r.raise_for_status()63 return [{"title": x["title"], "url": x["url"], "snippet": x["description"],64 "age": x.get("age")} for x in r.json()["web"]["results"]]6566def serper_search(q, n=5):67 r = requests.post("https://google.serper.dev/search",68 headers={"X-API-KEY": os.environ["SERPER_API_KEY"]},69 json={"q": q, "num": n}, timeout=20)70 r.raise_for_status()71 return [{"title": x["title"], "url": x["link"], "snippet": x["snippet"]}72 for x in r.json().get("organic", [])[:n]]73```7475## Quota discipline76771. Only HTTP 200 counts against real quota — log 429/5xx separately, never increment.782. Cache by normalized URL; identical repeat queries should never re-hit the API.793. Route cheap dev traffic to SearXNG; save metered keys for production paths.804. Track monthly usage per key in a state file; alert at 80% before exhaustion.815. Monthly refills reset on the 1st and do not roll over.8283## Pitfalls8485- Tavily: key in body, not header. `advanced` depth = 2× credits.86- Exa: no relevance scores returned; `/contents` costs extra beyond `/search`.87- Firecrawl: crawl jobs are async (job ID first, poll later) — not empty results.88- Brave: card required at signup now; summarizer/grounding endpoints bill faster than plain search.89- Google CSE: needs BOTH `key` and `cx`; quota resets midnight PT not IST.
Drvivek34