mathews-tom Tavily Search
Web search and URL content extraction powered by Tavily API. Returns clean, structured results optimized for AI consumption — no HTML noise, no cookie banners, no pagination.
When to Use Tavily vs Other Tools
| Situation | Tool | Reason |
|---|---|---|
| Need current information on a topic | search.mjs |
Tavily aggregates and ranks live results |
| Need news from the last N days | search.mjs --topic news --days N |
News mode filters recency |
| Have a specific URL, need full text | extract.mjs |
Extracts clean article content from URL |
| Need documentation for a library | search.mjs --deep |
Deep mode covers docs sites thoroughly |
| Need a static resource already known | direct fetch or read | Skip Tavily if no live lookup needed |
| Need multiple competing perspectives | search.mjs -n 10 |
Increase result count for breadth |
Scripts
Both scripts exist at scripts/ and require TAVILY_API_KEY in the environment.
scripts/search.mjs— query-based web searchscripts/extract.mjs— full content extraction from a URL
Search
node {baseDir}/scripts/search.mjs "query"
node {baseDir}/scripts/search.mjs "query" -n 10
node {baseDir}/scripts/search.mjs "query" --deep
node {baseDir}/scripts/search.mjs "query" --topic news
node {baseDir}/scripts/search.mjs "query" --topic news --days 3
Search Options
| Option | Default | Description |
|---|---|---|
-n <count> |
5 | Number of results to return (max: 20). Increase for broader coverage. |
--deep |
off | Advanced search mode — slower but more comprehensive. Use for complex research questions, technical deep-dives, or when standard results are insufficient. |
--topic <topic> |
general |
general for standard search; news for current events filtered by recency. |
--days <n> |
— | Only valid with --topic news. Limits results to articles published in the last n days. |
Extract Content from URL
node {baseDir}/scripts/extract.mjs "https://example.com/article"
Use when you already have a URL and need the full page content — Tavily strips navigation, ads, and boilerplate, returning structured article text.
Output Format
Search results return a JSON array. Each result contains:
[
{
"title": "Article or page title",
"url": "https://source.com/path",
"content": "Clean text snippet (AI-optimized excerpt)",
"score": 0.92
}
]
score is Tavily's relevance ranking (0–1). Higher is more relevant. When parsing results,
use content for summarization and url for attribution. Do not treat content as the full
article — use extract.mjs on the URL if complete text is needed.
Extract output returns a single object:
{
"url": "https://source.com/path",
"raw_content": "Full extracted page text..."
}
Error Handling
| Error | Cause | Resolution |
|---|---|---|
TAVILY_API_KEY not set |
Environment variable missing | Set TAVILY_API_KEY before running. Get a key at https://tavily.com |
401 Unauthorized |
Invalid API key | Verify the key value; regenerate at Tavily dashboard if needed |
429 Too Many Requests |
Rate limit hit | Reduce request frequency; free tier allows 1000 API credits/month |
Empty results array [] |
No matches found | Broaden or rephrase the query; try --deep for niche topics |
ETIMEDOUT / timeout |
Network or API latency | Retry once; --deep searches take longer (up to 30s) |
| Extract returns empty content | Page blocks scrapers | Use a different source or fall back to a manual fetch approach |
Limitations
- Rate limits: Free tier is 1000 API credits/month. Each search costs 1 credit;
--deepcosts more. - Content freshness: Results reflect Tavily's index; very recent content (last few hours) may not appear in
generalmode. Use--topic newsfor recency-sensitive queries. - Result quality: Tavily optimizes for AI-readable content but cannot guarantee factual accuracy. Verify critical claims from primary sources.
- Paywalled content: Articles behind paywalls return only the publicly accessible excerpt.
- Non-text content: PDFs, images, and media pages extract poorly or not at all.
- Language: Best results for English queries. Other languages are supported but result quality varies.