AI Fetch Web

ai-fetch-web

ai-fetch-web is a single bash script (no runtime other than bash + curl + jq + awk + base64) that wraps a fetch_web MCP server so any agent harness can use it without MCP client code.

Run ai-fetch-web --help in a shell for the authoritative reference. This file is the orientation doc.

When to use

• Default web access when ai-fetch-web is on $PATH. It replaces ad-hoc curl + "please parse this HTML" prompts and replaces harness-native fetch_web MCP calls where the MCP integration is unavailable or noisy.
• Reach for it for: web search, fetching a URL as clean markdown, batch-fetching multiple URLs in one call, extracting page fields by CSS selector, or capturing a screenshot.
• Do not use ai-fetch-web when the user is testing a specific HTTP behavior (headers, redirects, POST body shape), that is raw curl's job. ai-fetch-web hides response metadata by default.

Quickstart

# Search
ai-fetch-web search "rust 1.0 release" --limit 5

# Fetch one URL → clean markdown on stdout
ai-fetch-web fetch https://example.com

# Fetch many URLs in one server-side parallel batch
ai-fetch-web fetch-many https://a.example https://b.example

# Structured extraction (JSON on stdout)
ai-fetch-web extract https://shop.example --fields 'title:h1;price:.price'

# Screenshot (curl-style: -o PATH, or stdout if redirected)
ai-fetch-web screenshot https://example.com -o /tmp/page.png

# Ping the server / verify config
ai-fetch-web defaults

Subcommand reference

--fields short-form

--fields 'name:selector;name:selector' creates a {type: "value", selector: "..."} schema entry per pair. For list or table fields, or extras like attribute/format, pass --fields-file PATH with a full JSON schema:

{
  "products": {
    "type": "list",
    "selector": "main article.product",
    "fields": {
      "name": { "type": "value", "selector": "h2" },
      "price": { "type": "value", "selector": ".price" }
    }
  }
}

--json for programmatic callers

Every subcommand accepts --json. With it, stdout is the raw MCP result object (same shape as the server's tool response) instead of the rendered view. Use --json whenever you pipe to jq or store the response:

ai-fetch-web search "rust 1.0 release" --json \
  | jq -r '.content[].text'

ai-fetch-web metadata https://example.com --json \
  | jq -r '.content[0].text | fromjson | .openGraph'

The rendered default is designed to be parseable too (blocks + Key: value preludes), but --json is strictly better when you need a stable shape.

Configuration

ai-fetch-web has no built-in server URL. Set these env vars before use:

Verify configuration at any time:

ai-fetch-web defaults           # returns the server's defaults JSON
ai-fetch-web -v defaults        # and shows the config source + JSON-RPC trace on stderr

Limits and non-goals

• One request per invocation. No caching, no automatic retries. Wrap in until / jq / shell retry logic if you need that.
• No streaming. The server answers MCP JSON-RPC with a single SSE frame; ai-fetch-web reads it whole.
• No headless browser on the client side. All rendering (including screenshot) is done server-side; the CLI just ships bytes.
• Exit codes: 0 success, 1 tool/RPC error, 2 usage error (unknown flag, missing arg, bad selector format), 3 config missing or network/HTTP failure. Prefer checking $? over parsing stderr.