Web Fetch
Fetch one URL, prove it was allowed, extract the part the caller asked for, and
return that slice by digest with the provenance needed to trust it later.
What this skill does
web-fetch retrieves a single public URL,
extracts text, metadata, links, or a JSON document, and seals the result so a downstream step can
cite the fetch without re-fetching. It checks the final host against the
effective host policy before and after redirects, reads the page whole, and returns
the final URL, the HTTP status, a content_digest over the retrieved body, the
extracted slice, and a provenance block recording when it ran, every redirect
hop, and how many bytes it read. The digest identifies the original response body; the selected extraction is
returned in the native result. A digest alone is not a stored-body locator.
This is the primitive an agent reaches for when it has already decided which page
to read. The decision it makes easier is "can I read this page, and what did it
actually say", with the answer backed by a digest instead of a remembered
paraphrase. The calling agent decides which sources matter and synthesizes
across them; web-fetch retrieves exactly one source. By default it may follow
that source across any publicly routable host. Supply allowlist only when the
operator needs to narrow the initial request and every redirect to named hosts.
The built-in operation declares zero provider cost: it makes an unauthenticated
local GET and never performs payment or metered provider dispatch. Host/network
failures still need real transport evidence; they are not fetched content.
The request path is native Runx code. Skills do not supply a JavaScript HTTP
client: the runtime owns public-network enforcement, redirect admission,
timeouts, retries, the operator's page ceiling, response digests, and
receipt-safe evidence.
When to use this skill
- An agent has chosen a specific page and needs its content bound to a
content_digest so a later step can cite it without re-fetching.
- A research pass needs each source retrieved through one bounded public-network
fetch with a complete redirect chain and byte count.
- A review must later prove what a page said at fetch time.
- A follow-on governed operation needs one source extracted as
text, metadata, or links.
When not to use this skill
- To judge, rank, or synthesize across sources. That remains caller reasoning;
web-fetch retrieves exactly one source and refuses to reason over many.
- To reach a private, loopback, link-local, or otherwise non-public address.
- To reach a host outside an explicitly supplied
allowlist, including a host
reached only through a redirect.
- To write anything. The only scope is
web.read; there is no repo, file,
wallet, or send authority here.
- To archive an original response body for later retrieval.
content_digest
identifies the bytes read; this operation returns the selected extraction and
does not promise a separately retrievable raw-body artifact.
- To carry credentials, cookies, or custom authorization headers. Use the
appropriate authenticated provider capability for those reads.
Procedure
- Require
url at runner preflight. A missing target fails before the graph
starts with the exact missing input.
- Use
allowlist: ["*"] when the caller omits it, meaning any publicly
routable host. A caller-supplied allowlist narrows that default. Match the
URL host against the effective allowlist. On a miss, return policy_denied
before any network call, recording the attempted host and the allowlist it
was checked against.
- Fetch, following redirects, re-checking each redirect target's host against
the same
allowlist. A redirect that lands off-allowlist halts the fetch,
fails with policy_denied naming the hop that failed. The page is read
whole: a page larger than the operator's ceiling (RUNX_WEB_READ_MAX_BYTES,
64 MiB by default) fails with response_too_large instead of being cut.
- Compute
content_digest over the retrieved body.
- Extract per
extract: text (readable body text, default), metadata
(title, description, canonical, declared language, content type), links
(absolute hrefs found in the document), or json (the body as one JSON
document).
- Return the
provider_operation packet whose result holds the final URL,
status, digest, extracted slice, and provenance. A refused URL, a transport
failure, or a non-2xx answer fails the step; no result stands for a page
that was not read.
Edge cases and stop conditions
- Missing
url: fail runner preflight; do not start or seal an empty fetch.
- Omitted
allowlist: use ["*"]; private-network protection still applies.
- Host off the allowlist: stop with
policy_denied before any network call;
record the attempted host, not a response body (there is none).
- Redirect off the allowlist: halt the fetch, return
policy_denied naming
the hop that failed, and discard the partial body.
- Page past the ceiling: fail with
response_too_large; the operator
raises RUNX_WEB_READ_MAX_BYTES to read it. A partial page is never
returned.
- Raw-body identity: retain the selected extraction and original
content_digest; do not pass that digest to an artifact reader as if it were
a registered artifact locator.
- Authenticated source: stop and use the relevant provider read skill; this
public fetch does not accept credentials or custom headers.
Output schema
provider_operation:
data:
provider: web
operation: page.read
result:
final_url: string # URL after redirects, the one the digest is over
status: number # HTTP status of the final response
content_digest: string # digest of the body, algorithm prefix included
extract_mode: text | metadata | links | json
extracted: string | object | array # text, metadata object, hrefs, or the JSON document
provenance:
fetched_at: string # timestamp of the fetch
redirects: array # ordered hops, each re-checked against the allowlist
bytes: number # bytes read
The sealed runx.receipt.v1 carries the final URL, status, content_digest,
byte count, and the redirect chain. A denied host fails the step with
policy_denied before any request leaves. The receipt carries no header
values, no cookies, and no raw body beyond the digest.
Worked example
Input: url of the HTTP Semantics RFC, an allowlist of www.rfc-editor.org
and rfc-editor.org, and extract: text.
Output: the host matched the allowlist before the request left; no redirects;
status 200; content_digest is taken over the whole body; extracted holds
the readable text slice; provenance records fetched_at, an empty redirect
chain, and 184302 bytes.
The receipt seals with the final URL, status, digest, and byte count; no header
value reaches it.
Inputs
url (required): the single public HTTP(S) URL to fetch.
allowlist (optional): permitted hosts or host patterns for the URL and every
redirect target. Omit it for ["*"], meaning any publicly routable host.
extract (optional): text, metadata, links, or json. Defaults to
text.
1---2name: web-fetch3description: Fetch and extract one public web source, optionally restrict its hosts, and return the content by digest with full provenance.4metadata:5 source: https://skillmd.com/skills/runxhq/web-fetch6---78# Web Fetch910Fetch one URL, prove it was allowed, extract the part the caller asked for, and11return that slice by digest with the provenance needed to trust it later.1213## What this skill does1415`web-fetch` retrieves a single public URL,16extracts text, metadata, links, or a JSON document, and seals the result so a downstream step can17cite the fetch without re-fetching. It checks the final host against the18effective host policy before and after redirects, reads the page whole, and returns19the final URL, the HTTP status, a `content_digest` over the retrieved body, the20extracted slice, and a provenance block recording when it ran, every redirect21hop, and how many bytes it read. The digest identifies the original response body; the selected extraction is22returned in the native result. A digest alone is not a stored-body locator.2324This is the primitive an agent reaches for when it has already decided which page25to read. The decision it makes easier is "can I read this page, and what did it26actually say", with the answer backed by a digest instead of a remembered27paraphrase. The calling agent decides which sources matter and synthesizes28across them; `web-fetch` retrieves exactly one source. By default it may follow29that source across any publicly routable host. Supply `allowlist` only when the30operator needs to narrow the initial request and every redirect to named hosts.3132The built-in operation declares zero provider cost: it makes an unauthenticated33local GET and never performs payment or metered provider dispatch. Host/network34failures still need real transport evidence; they are not fetched content.3536The request path is native Runx code. Skills do not supply a JavaScript HTTP37client: the runtime owns public-network enforcement, redirect admission,38timeouts, retries, the operator's page ceiling, response digests, and39receipt-safe evidence.4041## When to use this skill4243- An agent has chosen a specific page and needs its content bound to a44 `content_digest` so a later step can cite it without re-fetching.45- A research pass needs each source retrieved through one bounded public-network46 fetch with a complete redirect chain and byte count.47- A review must later prove what a page said at fetch time.48- A follow-on governed operation needs one source extracted as49 `text`, `metadata`, or `links`.5051## When not to use this skill5253- To judge, rank, or synthesize across sources. That remains caller reasoning;54 `web-fetch` retrieves exactly one source and refuses to reason over many.55- To reach a private, loopback, link-local, or otherwise non-public address.56- To reach a host outside an explicitly supplied `allowlist`, including a host57 reached only through a redirect.58- To write anything. The only scope is `web.read`; there is no repo, file,59 wallet, or send authority here.60- To archive an original response body for later retrieval. `content_digest`61 identifies the bytes read; this operation returns the selected extraction and62 does not promise a separately retrievable raw-body artifact.63- To carry credentials, cookies, or custom authorization headers. Use the64 appropriate authenticated provider capability for those reads.6566## Procedure67681. Require `url` at runner preflight. A missing target fails before the graph69 starts with the exact missing input.702. Use `allowlist: ["*"]` when the caller omits it, meaning any publicly71 routable host. A caller-supplied allowlist narrows that default. Match the72 URL host against the effective allowlist. On a miss, return `policy_denied`73 before any network call, recording the attempted host and the allowlist it74 was checked against.753. Fetch, following redirects, re-checking each redirect target's host against76 the same `allowlist`. A redirect that lands off-allowlist halts the fetch,77 fails with `policy_denied` naming the hop that failed. The page is read78 whole: a page larger than the operator's ceiling (`RUNX_WEB_READ_MAX_BYTES`,79 64 MiB by default) fails with `response_too_large` instead of being cut.804. Compute `content_digest` over the retrieved body.815. Extract per `extract`: `text` (readable body text, default), `metadata`82 (title, description, canonical, declared language, content type), `links`83 (absolute hrefs found in the document), or `json` (the body as one JSON84 document).856. Return the `provider_operation` packet whose result holds the final URL,86 status, digest, extracted slice, and provenance. A refused URL, a transport87 failure, or a non-2xx answer fails the step; no result stands for a page88 that was not read.8990## Edge cases and stop conditions9192- **Missing `url`:** fail runner preflight; do not start or seal an empty fetch.93- **Omitted `allowlist`:** use `["*"]`; private-network protection still applies.94- **Host off the allowlist:** stop with `policy_denied` before any network call;95 record the attempted host, not a response body (there is none).96- **Redirect off the allowlist:** halt the fetch, return `policy_denied` naming97 the hop that failed, and discard the partial body.98- **Page past the ceiling:** fail with `response_too_large`; the operator99 raises `RUNX_WEB_READ_MAX_BYTES` to read it. A partial page is never100 returned.101- **Raw-body identity:** retain the selected extraction and original102 `content_digest`; do not pass that digest to an artifact reader as if it were103 a registered artifact locator.104- **Authenticated source:** stop and use the relevant provider read skill; this105 public fetch does not accept credentials or custom headers.106107## Output schema108109```yaml110provider_operation:111 data:112 provider: web113 operation: page.read114 result:115 final_url: string # URL after redirects, the one the digest is over116 status: number # HTTP status of the final response117 content_digest: string # digest of the body, algorithm prefix included118 extract_mode: text | metadata | links | json119 extracted: string | object | array # text, metadata object, hrefs, or the JSON document120 provenance:121 fetched_at: string # timestamp of the fetch122 redirects: array # ordered hops, each re-checked against the allowlist123 bytes: number # bytes read124```125126The sealed `runx.receipt.v1` carries the final URL, status, `content_digest`,127byte count, and the redirect chain. A denied host fails the step with128`policy_denied` before any request leaves. The receipt carries no header129values, no cookies, and no raw body beyond the digest.130131## Worked example132133Input: `url` of the HTTP Semantics RFC, an `allowlist` of `www.rfc-editor.org`134and `rfc-editor.org`, and `extract: text`.135136Output: the host matched the allowlist before the request left; no redirects;137status `200`; `content_digest` is taken over the whole body; `extracted` holds138the readable text slice; provenance records `fetched_at`, an empty redirect139chain, and `184302` bytes.140The receipt seals with the final URL, status, digest, and byte count; no header141value reaches it.142143## Inputs144145- `url` (required): the single public HTTP(S) URL to fetch.146- `allowlist` (optional): permitted hosts or host patterns for the URL and every147 redirect target. Omit it for `["*"]`, meaning any publicly routable host.148- `extract` (optional): `text`, `metadata`, `links`, or `json`. Defaults to149 `text`.