Golem Retry Policies TS

Configuring Semantic Retry Policies (TypeScript)

Golem provides a composable, per-environment retry policy system. Policies are evaluated against error context properties and can be defined in the application manifest, managed via CLI, or created/overridden at runtime from agent code using the SDK.

1. Define Retry Policies in the Application Manifest

Add retry policy definitions under retryPolicyDefaults in golem.yaml, scoped per environment:

retryPolicyDefaults:
  prod:
    http-transient:
      priority: 10
      predicate:
        and:
          - propEq: { property: "error-type", value: "transient" }
          - propEq: { property: "uri-scheme", value: "https" }
      policy:
        countBox:
          maxRetries: 5
          inner:
            jitter:
              factor: 0.15
              inner:
                clamp:
                  minDelay: "100ms"
                  maxDelay: "5s"
                  inner:
                    exponential:
                      baseDelay: "200ms"
                      factor: 2.0

    catch-all:
      priority: 0
      predicate: true
      policy:
        countBox:
          maxRetries: 3
          inner:
            exponential:
              baseDelay: "100ms"
              factor: 3.0

Policy Evaluation Order

When an error occurs, policies are evaluated in descending priority order. The first matching predicate's policy is applied. If no user-defined policy matches, the built-in default policy (3 retries, exponential backoff, clamped to [100ms, 1s], 15% jitter) is used.

Base Policies

Combinators

Predicates

Predicates are boolean expressions evaluated against error context properties. Compose with and, or, not:

• true / false — always/never match
• propEq — property equals a value
• propIn — property is one of a set of values
• propGte / propLt — numeric comparisons
• and / or / not — logical composition

Available Properties

Every retry decision happens in a specific context (an outgoing HTTP request, an HTTP response, a worker-to-worker RPC call, a trap from inside the guest, etc.). Each context only populates a subset of the property vocabulary below — a policy keyed on a property that is not present in the current context is silently skipped for that decision (it cannot apply there by definition).

Common to every context:

• verb — operation verb (HTTP method, RDBMS verb, RPC verb, or "trap" in the trap context)
• noun-uri — the resource URI (https://..., worker://..., kv://..., blobstore://..., dns://..., wasm://<function> for traps, golem://api, …)
• uri-scheme, uri-host, uri-port, uri-path — decomposed from noun-uri

Context-specific properties:

Practical consequence. A status-code-keyed policy (predicate: status-code in [...]) only fires for HTTP responses. The trap path does not see status-code and silently skips that policy — it does not error out. Likewise, a trap-type-keyed policy only fires from the trap path. Design one policy per context (or use or/and to make a policy explicitly match multiple contexts) rather than expecting a single policy to apply everywhere.

error-type values

• transient — transient transport failure (e.g. WASI HTTP error code, transient RDBMS error)
• http-status — HTTP response with a status code that matched a status-code-keyed policy

Status-code retries (opt-in)

Outgoing HTTP responses (including those returned by fetch()) now flow through the retry-policy machinery: when the response arrives, its status-code is exposed to predicates. A policy is only considered for status-code retries if its predicate (or the predicate inside a nested FilteredOn) explicitly references the status-code property. Catch-all policies — including the synthesized default and any user-defined "matches all" policy — are intentionally excluded so status-based retries remain strictly opt-in.

When a matching policy decides to retry, the rejected response is dropped, the request body is reconstructed from the oplog, and the request is re-sent — without needing atomically(...) and without the user-land throw pattern.

Eligibility rules (mirror inline transport retry):

• live execution (not replay or snapshotting)
• request body and trailers are reconstructible
• the HTTP method is idempotent, or assumeIdempotence was set on the outgoing request
• not inside an atomically(...) block — in v1 status retries are skipped inside atomic regions; the user-land throw still triggers atomic-region replay, which gives equivalent end-to-end behavior

Example status-code policy:

http-5xx-retry:
  priority: 20
  predicate:
    and:
      - propIn: { property: "status-code", values: [500, 502, 503, 504] }
      - propEq: { property: "uri-scheme", value: "https" }
  policy:
    countBox:
      maxRetries: 3
      inner:
        exponential:
          baseDelay: "200ms"
          factor: 2.0

fetch() interaction: fetch() resolves with a Response for HTTP 4xx/5xx statuses. If a status-code-keyed retry policy matches, the host transparently re-sends the request and the resolved Response will be the result of the last attempt — no extra atomically(...) is needed. The atomically(...) pattern below is still useful when the retry trigger is application-level (e.g. retry on a parsed body field rather than a status code), or to combine multiple side effects into one logical step.

2. SDK: Build and Apply Retry Policies at Runtime

Use @golemcloud/golem-ts-sdk to construct and apply retry policies from agent code:

import {
  Policy, Predicate, NamedPolicy, Props, Duration,
  withRetryPolicy,
} from '@golemcloud/golem-ts-sdk';

const policy = NamedPolicy.named(
  'http-transient',
  Policy.exponential(Duration.milliseconds(200), 2.0)
    .clamp(Duration.milliseconds(100), Duration.seconds(5))
    .withJitter(0.15)
    .onlyWhen(Predicate.eq(Props.errorType, 'transient'))
    .maxRetries(5),
)
  .priority(10)
  .appliesWhen(Predicate.eq(Props.uriScheme, 'https'));

Scoped Usage with withRetryPolicy

Apply a policy for a block of code — the previous policy is restored when the block exits:

withRetryPolicy(policy, () => {
  // HTTP calls in this block use the custom retry policy
  makeHttpRequest();
});

When the retryable condition is detected by application code after fetch returns, combine withRetryPolicy with atomically so retries re-send the request instead of replaying the already recorded response:

import {
  Duration,
  NamedPolicy,
  Policy,
  atomically,
  withRetryPolicy,
} from '@golemcloud/golem-ts-sdk';

const paymentRetry = NamedPolicy.named(
  'payment-retry',
  Policy.exponential(Duration.seconds(1), 2.0)
    .clamp(Duration.milliseconds(500), Duration.seconds(10)),
);

const payment = await withRetryPolicy(paymentRetry, () =>
  atomically(async () => {
    const response = await fetch('https://payments.example.com/charge', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ orderId, amount }),
    });

    if (!response.ok) {
      const body = await response.text();
      throw new Error(`payment failed: ${response.status} ${body}`);
    }

    return await response.json();
  }),
);

Policy Builder Methods

Build policies fluently from base policies:

// Exponential backoff clamped with jitter and max retries
Policy.exponential(Duration.milliseconds(200), 2.0)
  .clamp(Duration.milliseconds(100), Duration.seconds(5))
  .withJitter(0.15)
  .maxRetries(5)

// Periodic with time limit
Policy.periodic(Duration.seconds(1))
  .timeBox(Duration.seconds(60))

// Immediate retries then fall back to exponential
Policy.immediate()
  .maxRetries(3)
  .andThen(
    Policy.exponential(Duration.seconds(1), 2.0)
      .maxRetries(5)
  )

// Never retry (fail immediately)
Policy.never()

Predicate Builder Methods

// Match transient host-level failures
Predicate.eq(Props.errorType, 'transient')

// Match a property value
Predicate.eq(Props.uriScheme, 'https')

// Combine predicates
Predicate.and([
  Predicate.eq(Props.errorType, 'transient'),
  Predicate.eq(Props.uriScheme, 'https'),
])

3. Querying Retry Policies at Runtime

Use the query API to inspect active policies from agent code:

import { getRetryPolicies, getRetryPolicyByName } from '@golemcloud/golem-ts-sdk';

// List all active policies
const policies = getRetryPolicies();
for (const p of policies) {
  console.log(`Policy '${p.name}' priority=${p.priority}`);
}

// Get a specific policy by name
const policy = getRetryPolicyByName('http-transient');
if (policy) {
  console.log(`Found policy with priority ${policy.priority}`);
}

The returned NamedRetryPolicy has fields: name (string), priority (number), predicate (RetryPredicate), policy (RetryPolicy).

4. Live-Editing Policies via CLI

Retry policies can be managed at runtime without redeployment:

# Create a new policy
golem retry-policy create http-transient \
  --priority 10 \
  --predicate '{ "and": [{ "propEq": { "property": "error-type", "value": "transient" } }, { "propEq": { "property": "uri-scheme", "value": "https" } }] }' \
  --policy '{ "countBox": { "maxRetries": 5, "inner": { "exponential": { "baseDelay": "200ms", "factor": 2.0 } } } }'

# List all policies in the current environment
golem retry-policy list

# Get a specific policy by name
golem retry-policy get http-transient

# Update an existing policy
golem retry-policy update http-transient --priority 15

# Delete a policy
golem retry-policy delete http-transient

5. Default Retry Policy

When no user-defined retry policies are set, Golem activates a default catch-all:

• Name: default
• Priority: 0
• Predicate: true (matches everything)
• Policy: Up to 3 retries, exponential backoff (factor 3.0), delays clamped to [100ms, 1s], 15% jitter

Key Constraints

• Policies are defined per-environment — different environments can have different retry behaviors
• Policy names must be unique within an environment
• Higher priority policies are evaluated first; the first matching predicate wins
• withRetryPolicy is scoped — the policy is restored when the callback exits
• Inline retries (automatic transparent retries for transient network errors) happen before the policy system kicks in
• Changes made via CLI or REST API take effect immediately for running agents