Web Fetch

Web Fetch

Fetch one URL, prove it was allowed, extract the part the caller asked for, and return that slice by digest with the provenance needed to trust it later.

What this skill does

web-fetch retrieves a single public URL, extracts text, metadata, or links, and seals the result so a downstream step can cite the fetch without re-fetching. It checks the final host against the effective host policy before and after redirects, retrieves up to max_bytes, and returns the final URL, the HTTP status, a content_digest over the retrieved body, the extracted slice, and a provenance block recording when it ran, every redirect hop, and how many bytes it read. The body is referenced by digest; only the extracted slice is inlined.

This is the primitive an agent reaches for when it has already decided which page to read. The decision it makes easier is "can I read this page, and what did it actually say", with the answer backed by a digest instead of a remembered paraphrase. The calling agent decides which sources matter and synthesizes across them; web-fetch retrieves exactly one source. By default it may follow that source across any publicly routable host. Supply allowlist only when the operator needs to narrow the initial request and every redirect to named hosts.

The request path is native Runx code. Skills do not supply a JavaScript HTTP client: the runtime owns public-network enforcement, redirect admission, timeouts, retries, byte limits, response digests, and receipt-safe evidence.

When to use this skill

• An agent has chosen a specific page and needs its content bound to a content_digest so a later step can cite it without re-fetching.
• A research pass needs each source retrieved through one bounded public-network fetch with a complete redirect chain and byte count.
• A review must later prove what a page said at fetch time.
• A follow-on governed operation needs one source extracted as text, metadata, or links.

When not to use this skill

• To judge, rank, or synthesize across sources. That remains caller reasoning; web-fetch retrieves exactly one source and refuses to reason over many.
• To reach a private, loopback, link-local, or otherwise non-public address.
• To reach a host outside an explicitly supplied allowlist, including a host reached only through a redirect.
• To write anything. The only scope is net:allowlist; there is no repo, file, wallet, or send authority here.
• To inline a large raw body. The extracted slice is the payload; the full body lives behind content_digest.
• To carry credentials, cookies, or custom authorization headers. Use the appropriate authenticated provider capability for those reads.

Procedure

1. Require url at runner preflight. A missing target fails before the graph starts with the exact missing input.
2. Use allowlist: ["*"] when the caller omits it, meaning any publicly routable host. A caller-supplied allowlist narrows that default. Match the URL host against the effective allowlist. On a miss, return policy_denied before any network call, recording the attempted host and the allowlist it was checked against.
3. Fetch, following redirects, re-checking each redirect target's host against the same allowlist. A redirect that lands off-allowlist halts the fetch, returns policy_denied with the hop that failed, and discards partial bodies. Cap the read at max_bytes when set.
4. Compute content_digest over the retrieved body.
5. Extract per extract: text (readable body text, default), metadata (title, description, canonical, declared language, content type), or links (absolute hrefs found in the document).
6. Return fetch_result with the final URL, status, digest, extracted slice, and provenance. Flag truncated reads in provenance; never return a clipped read as if whole.

Edge cases and stop conditions

• Missing url: fail runner preflight; do not start or seal an empty fetch.
• Omitted allowlist: use ["*"]; private-network protection still applies.
• Host off the allowlist: stop with policy_denied before any network call; record the attempted host, not a response body (there is none).
• Redirect off the allowlist: halt the fetch, return policy_denied naming the hop that failed, and discard the partial body.
• Read clipped by max_bytes: flag truncated: true in provenance; the digest is over the bytes actually retrieved.
• Large raw body: never inline beyond the extracted slice; anything bigger than the requested view is reachable only through content_digest.
• Authenticated source: stop and use the relevant provider read skill; this public fetch does not accept credentials or custom headers.

Output schema

fetch_result:
  decision: ready | needs_agent | policy_denied | provider_error
  final_url: string            # URL after redirects, the one the digest is over
  status: number               # HTTP status of the final response
  content_digest: string       # digest of the retrieved body, algorithm prefix included
  extract_mode: text | metadata | links
  extracted: string | object | array   # string for text, object for metadata, array of hrefs for links
  provenance:
    fetched_at: string         # timestamp of the fetch
    redirects: array           # ordered host hops, each re-checked against the allowlist
    bytes: number              # bytes read
    truncated: boolean         # true when max_bytes clipped the read
  policy:
    allowlist_decision: allowed | denied
    attempted_host: string     # set on policy_denied
    allowlist_checked: array   # the hosts the request was checked against

The sealed runx.receipt.v1 carries the final URL, status, content_digest, byte count, the redirect chain, and the allowlist decision. It carries no header values, no cookies, and no raw body beyond the digest.

Worked example

Input: url of the HTTP Semantics RFC, an allowlist of www.rfc-editor.org and rfc-editor.org, extract: text, and max_bytes: 200000.

Output: decision: ready; the host matched the allowlist before the request left; no redirects; status 200; content_digest is taken over the retrieved body; extracted holds the readable text slice; provenance records fetched_at, an empty redirect chain, 184302 bytes, and truncated: false. The receipt seals with the final URL, status, digest, byte count, and the allowlist decision; no header value reaches it.

Inputs

• url (required): the single public HTTP(S) URL to fetch.
• allowlist (optional): permitted hosts or host patterns for the URL and every redirect target. Omit it for ["*"], meaning any publicly routable host.
• extract (optional): text, metadata, or links. Defaults to text.
• max_bytes (optional): cap on bytes read; a clipped read is flagged truncated in provenance.