tomevault-io Smart Fetch Tools
When to use which tool
| Need | Tool | When |
|---|---|---|
| Fetch a single URL | smart_fetch |
Articles, posts, docs, any page — try this first |
| Fetch multiple URLs | batch_smart_fetch |
Multiple URLs in one call, bounded concurrency |
| JS-heavy interactive sites | browser |
SPAs that need JavaScript to render content |
Sites and pages smart_fetch handles well
smart_fetch uses Defuddle for content extraction and wreq-js for browser-grade TLS fingerprinting. This combination works especially well on:
| Site / page type | What it extracts |
|---|---|
| X / Twitter posts | Tweet text via oEmbed; detects deleted/protected tweets |
| Reddit posts & threads | Post content + comment threads (use includeReplies) |
| YouTube | Page metadata, transcript extraction |
| GitHub | READMEs, issues, PRs, discussions — strips chrome, keeps code |
| Hacker News | Story content + comment threads |
| Substack / Medium | Full article text, author, publish date |
| Stack Overflow | Question + answers with code blocks |
| Wikipedia | Article body with infobox cleanup |
| Documentation sites | Code blocks, callouts, footnotes, math (MathML/KaTeX) |
| Blog posts & news | Schema.org metadata, clean main-content extraction |
| General web pages | Any HTML page — strips nav, sidebars, footers, ads |
Limitations — escalate to the browser tool for:
- JS-heavy SPAs — content that only appears after JavaScript execution
- Login-protected pages — no session/cookie management
- Interactive flows — anything needing clicks, form fills, or scrolling
smart_fetch
| Parameter | Type | Description |
|---|---|---|
url |
string (required) |
HTTP or HTTPS URL to fetch |
browser |
string |
TLS profile: chrome_145, firefox_147, safari_26, edge_145 |
os |
string |
OS profile: windows, macos, linux, android, ios |
headers |
Record<string,string> |
Custom HTTP headers |
maxChars |
number |
Max characters to return (default: 50000) |
timeoutMs |
number |
Request timeout in ms (default: 15000) |
format |
string |
Output: markdown (default), html, text, json, raw |
removeImages |
boolean |
Strip image references (default: false) |
includeReplies |
boolean or "extractors" |
Include comments/replies (default: "extractors") |
proxy |
string |
HTTP or SOCKS5 proxy URL |
Why smart_fetch over web_fetch
- TLS fingerprinting — impersonates real browsers at the TLS/HTTP2 level (JA3/JA4). Sites that return 403 or empty pages to plain HTTP clients often serve full content to smart_fetch.
- Better extraction — Defuddle removes more noise (nav, sidebars, ads, footers, social widgets) and keeps more signal (code blocks, footnotes, math, callouts, schema.org metadata).
- Richer metadata — returns author, publish date, site name, language, word count.
rawformat — returns the full unmodified server response (HTML, JSON, markdown, etc.) without extraction or truncation. Use when you need to parse the raw markup yourself.- No API key required — works out of the box.
batch_smart_fetch
| Parameter | Type | Description |
|---|---|---|
requests |
array (required) | Each item accepts the same params as smart_fetch |
- Default concurrency: 8 parallel requests (configurable via plugin config).
- Results are ordered matching the input array — labelled
[N/total]. - Individual failures don't fail the batch — each item has its own status.
Workflow escalation
smart_fetch— first choice for any URL.batch_smart_fetch— when you need multiple URLs at once.web_fetch— if smart_fetch is unavailable.- Browser tool — JS-heavy or login-protected pages only.
Automatic web_fetch fallback
When the smart-fetch plugin is enabled, it registers as a web fetch
provider. The built-in web_fetch tool will automatically use smart_fetch's
TLS-fingerprinted pipeline when its own Readability extraction fails — no
configuration needed.
Source: Thinkscape/agent-smart-fetch — distributed by TomeVault.