Golem Make HTTP Request Moonbit

Making Outgoing HTTP Requests (MoonBit)

Overview

MoonBit Golem agents use the SDK's @http package (golemcloud/golem_sdk/http) for outgoing HTTP requests. This package re-exports the WASI HTTP resource types (OutgoingRequest, Fields, IncomingResponse, …) and exposes the handle function that drives the outgoing handler.

All outgoing HTTP requests made from a Golem agent are automatically durably persisted — Golem records the request and response in the oplog, so on replay the response is read from the log rather than re-executing the network call.

Setup

Add the HTTP package to your agent's moon.pkg. Because the @http re-export only covers the resource types (not the Method / Scheme variant constructors), also import the underlying types package and the streams error type:

import {
  "golemcloud/golem_sdk/http" @http,
  "golemcloud/golem_sdk/interface/wasi/http/types" @types,
  "golemcloud/golem_sdk/interface/wasi/io/streams" @streams,
}

For UTF-8 encoding/decoding of bodies, also depend on the standard library encoders:

import {
  "moonbitlang/core/encoding/utf8" @utf8,
}

No WIT changes or binding regeneration is needed — the SDK already includes the HTTP imports.

Key API Facts (don't guess these)

The MoonBit HTTP API does not match TypeScript/Rust naming. Use exactly these constructors/methods:

Note: String::to_bytes() exists but returns UTF-16LE bytes — it is the wrong encoding for HTTP. Always use @utf8.encode for HTTP bodies and headers.

Helpers

These two helpers handle the byte/string conversions you need everywhere; copy them into your agent file:

///| Convert a UTF-8 String to a `FixedArray[Byte]` (the WASI `field-value` / body type).
fn str_to_bytes(s : String) -> FixedArray[Byte] {
  @utf8.encode(s).to_fixedarray()
}

///| Decode a `FixedArray[Byte]` as UTF-8 into a `String` (lossy — invalid sequences become U+FFFD).
fn bytes_to_str(bs : FixedArray[Byte]) -> String {
  @utf8.decode_lossy(Bytes::from_fixedarray(bs).view())
}

GET Request

///|
fn make_get_request(authority : String, path : String) -> String {
  // 1. Headers (empty).
  let headers = @types.Fields::fields()

  // 2. Build the request (defaults to GET).
  let request = @types.OutgoingRequest::outgoing_request(headers)
  let _ = request.set_scheme(Some(@types.Https))
  let _ = request.set_authority(Some(authority))
  let _ = request.set_path_with_query(Some(path))

  // 3. Finish the (empty) outgoing body.
  let body = request.body().unwrap()
  @types.OutgoingBody::finish(body, None).unwrap()

  // 4. Send.
  let future_response = @http.handle(request, None).unwrap()

  // 5. Wait for the response (FutureIncomingResponse + Pollable).
  let pollable = future_response.subscribe()
  pollable.block()
  // get() returns Option[Result[Result[IncomingResponse, ErrorCode], Unit]]
  let response = future_response.get().unwrap().unwrap().unwrap()

  // 6. Read status (UInt) and body.
  let _status = response.status()
  let incoming_body = response.consume().unwrap()
  let stream = incoming_body.stream().unwrap()
  let bytes = stream.blocking_read(1048576UL).unwrap() // up to 1 MiB
  stream.drop()
  let _ = @types.IncomingBody::finish(incoming_body)

  bytes_to_str(bytes)
}

POST with JSON Body

///|
fn make_post_request(
  authority : String,
  path : String,
  json_body : String,
) -> (UInt, String) {
  // 1. Headers — note: field values are FixedArray[Byte], not Bytes.
  let headers = @types.Fields::from_list([
    ("Content-Type", str_to_bytes("application/json")),
    ("Accept", str_to_bytes("application/json")),
  ]).unwrap()

  // 2. Build request and switch method to POST.
  let request = @types.OutgoingRequest::outgoing_request(headers)
  let _ = request.set_method(@types.Post)
  let _ = request.set_scheme(Some(@types.Https))
  let _ = request.set_authority(Some(authority))
  let _ = request.set_path_with_query(Some(path))

  // 3. Write the body.
  let body = request.body().unwrap()
  let output_stream = body.write().unwrap()
  output_stream.blocking_write_and_flush(str_to_bytes(json_body)).unwrap()
  output_stream.drop() // MUST drop the stream before finishing the body
  @types.OutgoingBody::finish(body, None).unwrap()

  // 4. Send and wait.
  let future_response = @http.handle(request, None).unwrap()
  let pollable = future_response.subscribe()
  pollable.block()
  let response = future_response.get().unwrap().unwrap().unwrap()

  // 5. Read response.
  let status = response.status()
  let incoming_body = response.consume().unwrap()
  let stream = incoming_body.stream().unwrap()
  let bytes = stream.blocking_read(1048576UL).unwrap()
  stream.drop()
  let _ = @types.IncomingBody::finish(incoming_body)

  (status, bytes_to_str(bytes))
}

Setting Headers

Fields is a WASI resource. Field values are FixedArray[Byte]:

// From a list of (name, value) pairs.
let headers = @types.Fields::from_list([
  ("Authorization", str_to_bytes("Bearer my-token")),
  ("Accept", str_to_bytes("application/json")),
  ("X-Custom-Header", str_to_bytes("custom-value")),
]).unwrap()

// Or construct empty and append.
let headers = @types.Fields::fields()
let _ = headers.append("Authorization", str_to_bytes("Bearer my-token"))
let _ = headers.append("Content-Type", str_to_bytes("application/json"))

Reading Response Headers

let response = future_response.get().unwrap().unwrap().unwrap()
let _status : UInt = response.status()
let resp_headers = response.headers()
// Fields::get returns Array[FixedArray[Byte]] (a single header may have multiple values).
let content_type_values = resp_headers.get("Content-Type")
let content_type : String? = match content_type_values.get(0) {
  Some(v) => Some(bytes_to_str(v))
  None => None
}

Setting Timeouts

RequestOptions is a resource constructed via request_options(). Timeouts are UInt64 nanoseconds:

let options = @types.RequestOptions::request_options()
let _ = options.set_connect_timeout(Some(5_000_000_000UL))         // 5 s
let _ = options.set_first_byte_timeout(Some(10_000_000_000UL))     // 10 s
let _ = options.set_between_bytes_timeout(Some(30_000_000_000UL))  // 30 s

let future_response = @http.handle(request, Some(options)).unwrap()

Retrying on non-2xx Responses

@http.handle resolves successfully when the server returns an HTTP response, even if the status is 500. From Golem's durability point of view, the recorded oplog entry already contains the 500 response, and any subsequent application-level error you produce based on the status is a separate failure after the side effect.

There are two ways to make Golem retry the HTTP call itself when a non-2xx response arrives.

Preferred: a status-code-keyed retry policy

Define a named retry policy whose predicate explicitly references status-code. When the response arrives the host transparently re-sends the request — no atomic block and no application-level error needed. The resolved response is the last attempt. See the golem-retry-policies-moonbit skill for the policy YAML / SDK shape.

POST/PUT/PATCH work out of the box. Idempotence mode defaults to true, so a plain POST /charge with a body is already eligible for status-code retry — there is no need to wrap the call in @api.with_idempotence_mode(true, ...). See golem-atomic-block-moonbit for details on when (rarely) to opt out with @api.with_idempotence_mode(false, ...).

The policy is defined once (in golem.yaml or via the SDK) and the call site is just a plain HTTP call — no extra wrapping is needed at the call site:

# golem.yaml — under retryPolicyDefaults / <env>:
http-5xx-retry:
  priority: 20
  predicate:
    propIn: { property: "status-code", values: [500, 502, 503, 504] }
  policy:
    countBox:
      maxRetries: 3
      inner:
        exponential:
          baseDelay: "200ms"
          factor: 2.0

If status retry "doesn't seem to work", load the golem-local-dev-server skill and look for the HTTP status retry skipped, reason: … debug line — it pinpoints why a particular request was not retried (e.g. BodyNotFinished, NotIdempotent, NoRetry).

Fallback: atomic block

When the retry trigger is application-level (e.g. retrying based on a parsed body field, or combining multiple side effects into one logical step), wrap the entire request, response-body read, and status check in an atomic block. The failed atomic block is not committed, so recovery re-executes the request instead of replaying the recorded failed response. See golem-atomic-block-moonbit.

Error Handling

Both @http.handle and FutureIncomingResponse::get can fail. get() returns Option[Result[Result[IncomingResponse, ErrorCode], Unit]] — None means not ready yet, the inner Err(()) means the response was already consumed:

///|
fn fetch_data(authority : String, path : String) -> Result[String, String] {
  let headers = @types.Fields::fields()
  let request = @types.OutgoingRequest::outgoing_request(headers)
  let _ = request.set_scheme(Some(@types.Https))
  let _ = request.set_authority(Some(authority))
  let _ = request.set_path_with_query(Some(path))

  let body = request.body().unwrap()
  @types.OutgoingBody::finish(body, None).unwrap()

  match @http.handle(request, None) {
    Err(error_code) => Err("Request failed: " + error_code.to_string())
    Ok(future_response) => {
      let pollable = future_response.subscribe()
      pollable.block()
      match future_response.get() {
        Some(Ok(Ok(response))) => {
          let status = response.status()
          let incoming_body = response.consume().unwrap()
          let stream = incoming_body.stream().unwrap()
          let bytes = stream.blocking_read(1048576UL).unwrap()
          stream.drop()
          let _ = @types.IncomingBody::finish(incoming_body)
          let body_str = bytes_to_str(bytes)
          if status >= 200 && status < 300 {
            Ok(body_str)
          } else {
            Err("HTTP " + status.to_string() + ": " + body_str)
          }
        }
        Some(Ok(Err(error_code))) =>
          Err("HTTP error: " + error_code.to_string())
        Some(Err(_)) => Err("Response already consumed")
        None => Err("Response not ready")
      }
    }
  }
}

Reading Large Response Bodies

InputStream::blocking_read returns up to the requested number of bytes and returns Err(@streams.Closed) when EOF has been reached. Read in a loop for arbitrary-size bodies:

///|
fn read_full_body(incoming_body : @types.IncomingBody) -> FixedArray[Byte] {
  let stream = incoming_body.stream().unwrap()
  let chunks : Array[FixedArray[Byte]] = []
  let mut done = false
  while !done {
    match stream.blocking_read(65536UL) {
      Ok(chunk) =>
        if chunk.length() == 0 {
          done = true
        } else {
          chunks.push(chunk)
        }
      Err(@streams.Closed) => done = true
      Err(e) => abort("stream read failed: " + e.to_string())
    }
  }
  stream.drop()
  let _ = @types.IncomingBody::finish(incoming_body)

  // Concatenate.
  let total = chunks.iter().fold(init=0, fn(acc, c) { acc + c.length() })
  let result : FixedArray[Byte] = FixedArray::make(total, b'\x00')
  let mut offset = 0
  for chunk in chunks {
    chunk.blit_to(result, len=chunk.length(), src_offset=0, dst_offset=offset)
    offset += chunk.length()
  }
  result
}

Complete Example in an Agent

///| An agent that fetches data from an external API.
#derive.agent
pub(all) struct DataFetcher {
  base_url : String
  mut last_result : String
}

///|
fn DataFetcher::new(base_url : String) -> DataFetcher {
  { base_url, last_result: "" }
}

///| Fetch data from the configured API endpoint.
pub fn DataFetcher::fetch(self : Self, path : String) -> String {
  let headers = @types.Fields::from_list([
    ("Accept", str_to_bytes("application/json")),
  ]).unwrap()

  let request = @types.OutgoingRequest::outgoing_request(headers)
  let _ = request.set_scheme(Some(@types.Https))
  let _ = request.set_authority(Some(self.base_url))
  let _ = request.set_path_with_query(Some(path))

  let body = request.body().unwrap()
  @types.OutgoingBody::finish(body, None).unwrap()

  let future_response = @http.handle(request, None).unwrap()
  let pollable = future_response.subscribe()
  pollable.block()
  let response = future_response.get().unwrap().unwrap().unwrap()

  let incoming_body = response.consume().unwrap()
  let stream = incoming_body.stream().unwrap()
  let bytes = stream.blocking_read(1048576UL).unwrap()
  stream.drop()
  let _ = @types.IncomingBody::finish(incoming_body)

  let result = bytes_to_str(bytes)
  self.last_result = result
  result
}

Calling Golem Agent HTTP Endpoints

When making HTTP requests to other Golem agent endpoints, the request body must match the Golem HTTP body mapping convention: non-binary body parameters are always deserialized from a JSON object where each top-level field corresponds to a method parameter name. This is true even when the endpoint has a single body parameter.

For example, given an agent endpoint:

#derive.endpoint(post = "/record")
pub fn RecorderAgent::record(self : Self, body : String) -> Unit { ... }

The correct HTTP request body is:

// ✅ CORRECT — JSON object with parameter name as key
let json_body = "{\"body\": \"hello\"}"

// ❌ WRONG — raw string does NOT match Golem's JSON body mapping
let json_body = "\"hello\""

Rule of thumb: If the target endpoint is a Golem agent, always send Content-Type: application/json with parameter names as JSON keys.

Resource Lifecycle

WASI HTTP uses resource handles that must be dropped in the correct order:

1. OutputStream must be dropped before calling OutgoingBody::finish.
2. OutgoingBody must be finished (@types.OutgoingBody::finish(body, None)), not just dropped, to signal the body is complete.
3. InputStream must be dropped before calling IncomingBody::finish.
4. IncomingBody must be finished to signal you're done reading.

Dropping a resource out of order will cause a trap.

Key Constraints

• All HTTP types are WASI resources with strict ownership and drop ordering.
• Field values (field-value) are FixedArray[Byte], not Bytes — b"..." literals are Bytes and do not type-check against Fields::append / Fields::from_list. Convert with @utf8.encode(s).to_fixedarray() (or b"...".to_fixedarray() for ASCII literals).
• String::to_bytes() returns UTF-16LE bytes — use @encoding/utf8.encode for HTTP, never String::to_bytes().
• String::from_utf8_lossy does not exist — use @utf8.decode_lossy(Bytes::from_fixedarray(arr).view()).
• The constructor for OutgoingRequest is outgoing_request(fields); for Fields it is fields(); for RequestOptions it is request_options(). There is no ::new on any of these.
• Method and Scheme variant constructors live in @types, not @http (@http only re-exports the type aliases). Use @types.Post, @types.Https, etc.
• request.method_() (with trailing underscore) is the getter — method is a reserved keyword.
• blocking_read reads up to the requested number of bytes — for large responses, read in a loop until you observe an empty chunk or Err(@streams.Closed).
• HTTP requests are automatically durably persisted by Golem — no manual durability wrapping is needed.
• Method defaults to Get when constructing an OutgoingRequest; only call set_method for non-GET requests.