Understanding Durable Execution
The worker executor runs WASM agents whose progress survives the loss of the process running
them. Read this guide alongside golem-worker-executor/src/ and golem-worker-executor-tests/tests/; when code
and guide disagree, the code wins and the guide needs a fix. Combined verification of
the instrumentation removal and adapted tests is pending. The scoped AGENTS.md files under
golem-worker-executor/ and golem-worker-executor-tests/ state the mandatory rules; this skill explains the mechanisms.
Deeper material lives in reference/: timelines.md (worked oplog timelines), crash-matrix.md
(what recovery does for each crash window), testing-patterns.md (tests that fail under a wrong
model), streams.md (durable streams and streaming invocations), tools.md (tool invocations
and entity bodies), retries.md (in-function versus trap-based retries), and
filesystem-inspection.md (exact-path live reads, shared scheduling and generation-pinned production).
Three axioms
Replay is deterministic by construction. The guest is a deterministic function of its guest-observable host inputs. The executor records every nondeterministic host interaction in the oplog and feeds the recorded result back during replay. If a replaying guest asks for a different host call than the one recorded next, the executor has a bug (missing recording, wrong reconstruction, wrong completion association). It is never "normal nondeterminism". The host side is not deterministic — completions are tokio futures and finish in varying order — but replay reproduces the guest-visible completion order from the recorded
CompletionDeliveredmarkers, so the guest sees the same inputs in the same order. Owner:durable_host/replay_state/claims.rs(a missingStartmatch while replay is active is a strict divergence error, except for a crash prefix whose remaining history belongs to other owners; see "Entry ownership". Operator-reverted entries are excluded). The axiom is about replaying the same execution contract; replaying old history against a changed component during anAutomaticupdate may legitimately diverge, and that is reported asFailedUpdate, not tolerated.The resident runtime is disposable. The Wasmtime
Store, the worker task, the executor process, sockets, channels and subscriptions can vanish while work is pending. Suspension, eviction, restart and crash must all be recoverable the same way, on the same executor: throw the instance away, build a newStore, replay the oplog, continue. Lifecycle hints (Suspend,Interrupted,Restart) change status and scheduling policy, not the recovery mechanism. Losing the shard does not reconstruct in place: a durable agent's oplog asserts this executor's shard epoch whenever it appends or deletes entries in storage, and once another executor holds the shard, the write is refused (OplogError::Fenced) instead of accepted. The agent is then retired for its lost shard (InterruptKind::ShardLost): that generation is stopped, dropped from this executor and never restarted, and the executor that now holds the shard builds a newStoreand replays — the new owner, or this executor again if the shard came back to it at a higher epoch. See "Resharding, revocation and the oplog epoch fence" below andcrash-matrix.md. Owner:worker/invocation_loop.rs::run(outer loop: create instance → recover → run → suspend/retry) anddurable_host/mod.rs::prepare_instance.Durable agent RPC is exactly-once at the logical callee-invocation level. Dispatch attempts (replay, transport retry, incomplete-call repair) share a durable key. An executing target persists one invocation and result; read-only cache hits/followers persist neither. Attempts are not executions. Owner:
durable_host/rpc/mod.rs(IdempotencyKeyPosition::derive(start_index)),worker/mod.rs::enqueue_worker_invocation_with_effect(lookup_invocation_resultdedupe before appendingPendingAgentInvocation).
┌──────────────┐ host call ┌──────────────────┐ append ┌──────────────┐
│ live guest │──────────▶│ durable_host fn │───────▶│ oplog │
│ (Store #1) │◀──────────│ Start…End…Deliver│ │ (persistent) │
└──────────────┘ result └──────────────────┘ └──────┬───────┘
✕ Store #1 disappears (suspend / evict / crash / restart) │
┌──────────────┐ host call ┌──────────────────┐ read │
│ replay guest │──────────▶│ claim Start, │◀────────┘
│ (Store #2) │◀──────────│ resolve End, │
└──────────────┘ recorded │ deliver at marker│
└────────┬─────────┘
cursor exhausted│ + reconstruction validated
▼
publish Live → new work appends again
Durable versus resident state
| Durable (authoritative) | Resident (disposable, derived) |
|---|---|
Oplog entries and their payloads (golem-common/src/base_model/oplog/mod.rs) |
Wasmtime Store, instance, linear memory |
PendingAgentInvocation / AgentInvocationStarted / AgentInvocationFinished |
Worker.queue: VecDeque<QueuedWorkerInvocation>, event subscriptions |
| Idempotency keys and recorded invocation results | hydrated_invocation_results cache, read-only cache |
Durable-call Start/End/Cancelled and CompletionDelivered/CompletionDiscarded markers |
DurableCallSession, ReplayableOneshot, spawned store tasks |
PendingUpdate / SuccessfulUpdate / FailedUpdate, Snapshot hints, snapshot blobs |
In-flight update decision, loaded snapshot bytes |
| Validated operator revert/fork cuts and retained history | Replay cursor and runtime handles |
Agent status folded from the oplog (worker/status.rs) |
Status cache / checkpoints (baselines only) |
Persisted promises and scheduler actions (WakeupScheduler), RunningWorkers recovery index |
Tokio timers, in-memory wake channels |
Anything in the right column may be rebuilt from the left column at any time; a design that needs
something from the right column to survive a restart is wrong. Sockets and other OS resources are
recreated, not preserved (durable_host/sockets, durable_host/http); the application protocol
must tolerate reconnect or explicitly handle loss. Replayed websocket handles use their connect-time
reconstruction policy under a per-handle coordination gate, so concurrent calls cannot reconnect
one handle twice or replace a terminal session-loss outcome (see
"Concurrency and guest completion delivery"). Durable liveness also needs rediscovery: a timed
wake is persisted as a scheduler action before automatic suspension (worker/suspension.rs::RuntimeStore::drive,
WakeupScheduler::sleep_until), and the shard-keyed RunningWorkers index is updated
synchronously so a crash/reshard can enumerate workers with pending work
(worker/status_flusher.rs). The status blob cache is an asynchronously flushed baseline only.
Status, clean-checkpoint, invocation-result-index, and durable-stream-index cache namespaces include
the current Create.instance_id (AgentFingerprint). Readers select them only after resolving the
authoritative Create entry, while resident workers carry that fingerprint directly. A delayed
writer from a deleted incarnation therefore cannot mix fields into its replacement's derived state.
These namespaces have a renewable expiry and are rebuilt from the oplog after expiry or cache loss;
metadata compare-and-mutate operations make each multi-field publication atomic. The flat cached
agent mode only chooses which oplog namespace to probe first and never proves existence or identity.
RunningWorkers entries also include the fingerprint, so shard recovery admits only the recorded
incarnation and removes only an exact stale member. A member is stale when its agent has no oplog,
or when the oplog of the agent belongs to another incarnation (stale_reason in
services/worker.rs). The recovery scan (WorkerService::get_running_workers_in_shards(on_stale))
and the restart after it (WorkerService::remove_if_stale) request delete_all_snapshots for the
dead incarnation of each stale member before they remove the member. A removal that fails leaves
the member, and the next scan requests the delete again. A delete request that a full clean-up
queue refuses is counted as a leaked clean-up: the scan still removes the member, and no later
scan asks again.
Scheduled Invoke, Resume and CompletePromise actions carry the target's fingerprint and
use WorkerActivator::enqueue_exact_existing. It checks identity before cold hydration and
again on the returned worker, covering cached and shared acquisition. Missing or mismatched
incarnations are logged and acknowledged without creation; acquisition, delivery and startup
errors remain retryable. Timer and quota producers capture their resident worker's fingerprint.
Scheduled promise completion acquires that worker, takes the local oplog lifecycle guard,
rechecks the stored identity, and persists/notifies through complete_without_wakeup. It releases
the guard before starting the pinned worker, and retries startup even when completion was already
persisted. Duplicate completion notifies with the original stored payload. Ordinary promise
completion still activates the worker. This guard serializes local deletion/recreation, not
cross-executor promise writes: promise storage still uses agent ID and oplog index. Deferred
InvokeEphemeral retains its separate creation path.
Two persistence steps matter for every crash window: append puts an entry in the oplog
buffer; commit makes it recoverable (commit_oplog_and_update_state(CommitLevel)). The
buffer is a performance trade-off (no storage round trip per append). CommitLevel
(services/oplog/mod.rs) says how strict a commit is: Always waits for durable storage;
Deferred still waits for durable oplogs but lets an ephemeral oplog return after ordered writer
handoff (bounded queue backpressure may wait); DurableOnly waits only for durable agents and
remains a no-op for ephemeral agents. Explicit protocol barriers and invocation acceptance still
use their existing storage guarantees; this is not a global weakening of commit semantics.
Successful invocation completion has a narrower contract. It appends
AgentInvocationFinished, then waits for the state actor's commit receipt before notifying
waiters. Durable completion uses Always, so that receipt follows storage. Ephemeral completion
uses Deferred, so its receipt proves ordered writer handoff, not storage. Neither mode waits for
the status fold: the actor retains that fold after acknowledging completion. A later actor-FIFO
status read that requires freshness waits behind the queued fold; the separately persisted status
cache remains asynchronous. Admission and authority jobs continue to await their complete folds,
and the RunningWorkers recovery index remains synchronously flushed by the status actor.
The unchanged synchronous Create write preserves an ephemeral agent's identity before execution;
after executor loss, that identity reconstructs an observation-only owner, never a fresh execution.
A commit can also be refused. A durable agent's primary oplog, opened while this executor holds
the agent's shard, asserts that shard epoch whenever it appends or deletes entries in storage — an
explicit commit, a threshold flush, a deletion — inside the storage transaction. Storage refuses the write with
OplogError::Fenced unless the record it holds names that epoch: a newer epoch means another
executor took over, and no record at all refuses too. An append only buffers, so it meets the check when the buffer is committed.
commit_oplog_and_update_state and add_and_commit_oplog surface the refusal rather than
swallowing it, so a refused PendingAgentInvocation commit is not acknowledged as accepted and a
refused AgentInvocationFinished commit is not published to waiters. The first refusal latches:
every later add or commit on that handle is refused locally, without reaching storage, and the
agent is retired for its lost shard instead of retried. The refusal travels as a typed error
(OplogError::Fenced, StreamStoreError::Fenced, SessionError::Fenced,
WorkerExecutorError::OplogFenced) to whoever acts on it; nothing re-reads the latch to
reinterpret a failure that already happened.
The shared direct and accessor durable-call start paths commit WriteRemote with
CommitLevel::DurableOnly after appending the actual Start and before returning the session
that permits the effect. This preserves preceding guest observations, including random inputs
used in a guest-supplied idempotency key. A refused commit prevents dispatch. The accessor path
releases the owner-wallet boundary before awaiting the commit. Completion still commits the End.
Snapshot calls append nothing; the barrier does not force storage for ephemeral agents.
The fence is checked at the commit, not atomically with the external effect: a shard can still
move after a successful commit, so the receiver must deduplicate retries. Socket sends,
DemandStream parents, and RDBMS protocol owners retain their explicit barriers. Oplogs that
assert no epoch are never refused: a handle opened before this executor has an assignment, and
fork stages and their publication. The archive transfer is fenced like the primary oplog (see
"Resharding, revocation and the oplog epoch fence" below), and so is an ephemeral agent's oplog,
which writes only through the compressed archive levels. The blob archive layer is fenced
through its manifest in indexed storage, since blob storage has no conditional write. One write
stays outside the fence: blob uploads of large payloads.
worker/state_actor.rs::commit_and_update_state samples the appended tip before its explicit
commit and ignores receipt entries already folded into the published status. Primary/ephemeral
threshold flushes and replica waits commit outside an explicit commit. After such a commit the
primary oplog calls its auto-commit listener (Oplog::set_auto_commit_listener, held weakly),
which queues one coalesced StatusJob::FoldAutoCommits. That job commits nothing: it folds the
entries returned by CommitLevel::ReportOnly, so status.oplog_idx follows the committed index
during a long invocation, while a buffered append below the threshold stays invisible until a
commit. The primary oplog keeps the entries of its latest auto commits in memory until that
report, which normally follows at once, up to oplog.max_retained_auto_commit_batches batches;
beyond it the report reads the oldest unreported entries back from storage. An ephemeral oplog
calls no listener and reports nothing for ReportOnly: ephemeral agents fold at the explicit
commit that ends each invocation. The job wakes the invocation loop only when a field other than the index changed; a
caller of add_and_commit_oplog_internal sends its own wakeup after every successful commit,
because the fold of a threshold commit can take its entry first. As the status follows the
oplog, each status-cache write catches up the stream-session and invocation-result indexes over
only the entries since the previous write. Even an empty receipt may still hide a committed
suffix. Ephemeral threshold flushes hand batches to a bounded asynchronous
writer; read snapshots include both handed-off entries and the buffered tail after the persisted
writer watermark. Up to 32 batch receipts are retained for status folding. If that cap creates a
receipt gap, exceptional catch-up forces storage only after the completion receipt has already
acknowledged the caller, then reads the missing range. Otherwise, unless the remaining receipt is
exactly the contiguous suffix after the last published index, the status actor catches up with
status::try_fold_status_from: committed storage is read in bounded chunks, external
StreamSession payloads are hydrated, and the result is published once. This avoids retaining an
unbounded auto-flushed tail and adds neither oplog entries nor a protocol change. Gap recovery
conservatively invalidates authority snapshots after the fold. This is status reconstruction, not
replay tolerance.
Component map
| Area | Files | Responsibility |
|---|---|---|
| Oplog model | golem-common/src/base_model/oplog/{mod.rs,oplog_macro.rs} |
Entry kinds, is_hint(), Start/End pairing rules |
| Replay cursor | durable_host/replay_state/{mod.rs,cursor.rs,claims.rs} |
Single cursor lock, speculative read vs commit, identity claims, Replaying→Settling→Live |
| Durable calls | durable_host/concurrent/{mod.rs,call.rs,delivery.rs,replay.rs} |
DurableCallSession state machine, drop policies, completion delivery tokens |
| Durability guard | durable_host/durability.rs |
begin_durable_function/end_durable_function, DurableFunctionType, in-function retry |
| Host context | durable_host/mod.rs, durable_host/recovery.rs |
prepare_instance, replay, switch_to_live, PendingReplayToLive, IdempotencyKeyPosition, snapshot boundary checks |
| Tail work | durable_host/tail_work.rs |
Keeps store tasks running until no durable work is active before AgentInvocationFinished |
| RPC | durable_host/rpc/mod.rs |
Key derivation, first dispatch vs MayExist, replay claims, ephemeral phantom identity |
| Worker | worker/{mod.rs,invocation_loop.rs,lifecycle.rs,instance/mod.rs,status.rs} |
Queue persistence, dedupe, interrupt/resume/update decisions, eviction, retry decisions |
| Owner wallet | worker/owner_wallet.rs, worker/instance/mod.rs |
Generation-bound live wallet, recorded replay baseline and shared cursor installation |
| Cut points | worker/cut_point.rs |
Validates operator cuts; calls recover from the exact retained prefix |
| Durable streams | durable_host/{durable_stream/mod.rs,durable_session/mod.rs,stream_session.rs,stream_bus.rs,stream_transport.rs,schema_value_stream.rs} |
Producer-oplog stream records, consumer session journal, exactly-once item delivery, protocol terminals |
| Tool invocations | durable_host/tool/{mod.rs,streams.rs,boundary.rs,operation/mod.rs,attachment.rs}, durable_host/entity.rs, worker/{entity_slot.rs,owner_lane.rs,entity_invocation.rs,instance/mod.rs} |
Per-admission authorization, inherited invocation scope, pinned middleware plans, typed boundaries, owner-oplog entity bodies, shared replay state, operation settlement |
Worker lifecycle and reconstruction
OwnerExecution installs the replay cursor and live wallet together in OwnerReplayGeneration.
Startup seals old append tasks, takes the card boundary, commits the horizon and selects the wallet
baseline before install_replay_generation_locked joins prior owner cleanup and begins the new
tool/deferred-admission generation. That installer is the single reset point. The earlier resident
attempt counter does not reset owner execution. After wallet installation succeeds, the loop marks
the owner as started so the next resident receives a new resource-release observation scope,
including when later context construction fails. Lifecycle markers do not use that bookkeeping.
Worker (worker/mod.rs) receives invocations, persists them, and processes them in order. The
run loop (worker/invocation_loop.rs::run) is:
┌─────────────────────────────────────────────────────────────────────┐
│ outer loop (one iteration = one instance attempt) │
│ monthly admission ──▶ create Store + instance ──▶ prepare_instance │
│ durable agent: handle PendingUpdate ▸ try snapshot ▸ replay │
│ ephemeral: constructor-only replay, append Restart │
│ ──▶ inner loop: pop durable queue, run invocation, persist Finished │
│ ──▶ instance ends (idle / interrupt / trap / suspend) │
│ ──▶ RetryDecision: Immediate | Delayed | ReacquirePermits | None | TryStop │
└─────────────────────────────────────────────────────────────────────┘
Only prepare_instance returning Ok(None) completes filesystem replay and reconstruction,
activates the resident generation, and enters the inner loop. Ok(Some(decision)) is an aborted
preparation, including Some(RetryDecision::None). PreparedAgent::finish seals that reconstructing
filesystem without drainage and retains the runtime for the ordinary teardown and retry tail.
The unload continuation drops the Store before it drains the sealed filesystem, closes the execution
window, releases the permit, and deletes the filesystem. A P2 output stream can retain a pending
FilesystemCall after interruption cancels only its borrowed readiness future; draining replay
with that Store still alive would wait on a call that only Store destruction can detach.
The original retry decision, later interrupt arbitration, ephemeral fail-stop conversion, and
cleanup-failure handling remain in the common tail.
DurableWorkerCtx::replay (durable_host/recovery.rs) loops get_oplog_entry_agent_invocation_started, replays
each recorded invocation in InvocationMode::Replay, and when there is no further
AgentInvocationStarted it switches to live. Ephemeral owners replay at most one recorded
AgentInitialization, and only when their resolved owner is a typed agent. Component-baseline
tool owners instantiate the deployed component but never queue or replay an agent constructor.
Constructor-only replay covers restarts before the accepted invocation ran. An ephemeral agent
never reconstructs in process once its Store has run: on_invocation_failure turns traps that
would restart it into terminal failures and notifies the caller. Any other Immediate, Delayed,
or ReacquirePermits decision reaching the outer loop becomes a terminal interruption plus archive,
without replacing an already published outcome. An active Suspend also becomes a terminal
EphemeralCannotSuspend failure after interrupt acknowledgement and monthly-exhaustion conversion,
so a lowered memory limit or failed monthly monitor cannot leave a method suffix for constructor-only
replay. Monthly capacity exhaustion retains its specific monthly error, and a monthly-policy outage
keeps its Monthly enforcement unavailable reason. Idle and pre-method suspension remain safe:
ordinary ephemeral agents keep their outer-loop TryStop unchanged;
ephemeral external-tool owners also treat it as terminal. Startup retries before a Store exists
remain unchanged. A started method never runs again live.
Replay starts from the chosen snapshot baseline
(see Snapshots and updates), not necessarily from OplogIndex::INITIAL. A new replay generation
scans the replayed range for recovery metadata (delivery markers, guest stream observations and
tool attachment indexes) before it resolves any call. A snapshot start scans only the entries
after the snapshot (replay_state/cursor.rs::recovery_metadata_floor): no durable construct
spans a snapshot, so nothing after it refers to metadata before it. A start without a snapshot
replays the whole history and scans it. Interruption kinds
(Worker::set_interrupting): Interrupt stays interrupted, Restart is a simulated crash with
automatic recovery, Suspend unloads and resumes on demand — each of these three reconstructs on
this same executor. ShardLost does not: it means this executor lost the agent's shard (a
revoked/reassigned shard, or an oplog write refused on the shard epoch), and instead of
reconstructing, that generation is retired — stopped without writing its status, dropped from
this executor and never restarted — and the executor that now holds the shard reconstructs it.
Eviction (EvictionClass::{LoadedIdle, WarmRunnable}) never unloads a
worker that is executing or holds non-durable in-memory work. A WarmRunnable eviction starts the
worker again in a spawned task (stop_if_evictable_with_outcome): the callers of its pending
invocations already started it and only wait, and the start waits for memory admission. Ephemeral agents are fail-stop:
reconstructed_ephemeral rebuilds only for observation and result lookup, "but the instance must
never be started again" (worker/mod.rs, INACTIVE_EPHEMERAL_AGENT_ERROR).
Monthly admission, interruption, and recovery
Inherit Worker acceptance, typed signals, invocation-loop cleanup and ordinary reconstruction; record independent defects and unverified combinations separately from the bounded delivery claim.
Monthly policy uses the shared account AtomicResourceEntry in services/resource_limits.rs.
Registry revisions arrive through the existing refresh path; stale refreshes must not restore
older policy. Admission in worker/invocation_loop.rs checks the latest cached capacity before
instance creation, replay resumption, and invocation. Reconstruction must pass those checks too;
a resume or queued invocation cannot bypass an exhausted applicable cap.
WorkAvailable and InternalStatusChanged are wake notifications, not work reservations. Before
reacquiring an idle concurrent-agent permit, the loop checks resident requests, pending interrupts,
detached status, durable pending work and snapshot policy. A notification whose work already drained
does not reopen a billing window. Detached status still wakes ordinary repair before work selection.
The loaded compute-admission test queues two real readiness requests while the actual Finished
append is withheld by the oplog dependency. An awaited filesystem update joins the earlier
notifications on the same command queue. Check queued-work results and accounting;
this no longer measures the number of execution-window acquisitions.
During guest execution, Context::ensure_fuel, the FuelManagement implementation in
workerctx/default.rs, calls ensure_monthly_resource_capacity to check enabled live compute,
then monthly memory and the agent's matching storage class. worker/instance/mod.rs calls it from the
existing Wasmtime epoch callback, and worker/invocation.rs checks before guest calls. Compute
and memory exhaustion apply account-wide; durable storage applies only to durable agents and
ephemeral storage only to ephemeral agents. Durable exhaustion requests suspension and unload
with RetryDecision::TryStop. Ephemeral exhaustion fails the invocation without retry, using
EphemeralFuelExhausted for compute or EphemeralCannotSuspend for memory/storage. Production
hard-limit execution does not borrow local ephemeral overdraft. Enforcement infrastructure
failure is distinct from exhausted capacity and must not be reported as an ordinary cap hit.
Capacity can return after a UTC month reset, a grant or plan increase above current usage, or
application of an allowOverage revision. Subsequent demand uses the same admission and
reconstruction path, not a second recovery system. Mode changes reach running agents through
refresh within five minutes. Idle refresh eligibility reserves one batch interval for the next
scheduled tick, so an account initialized just after a tick does not wait an extra interval.
The batch caches desired per-agent filesystem limits without awaiting a resident invocation's
filesystem-command reply. Each account has one latest-value queue and a serialized consumer that
uses the existing target callback. The Worker invocation loop still owns quota application and its
reply. A pending physical update cannot stop later monthly-policy refreshes or other accounts.
Queued newer limits cannot be overwritten in the cache by an older completion. Failures stay logged
and a later ordinary refresh can retry the same limit. Batch completion does not prove physical
quota application; explicit awaited filesystem-update callers retain their completion/error contract.
Usage attributed to an applied allowOverage revision remains eligible for excess billing until
hardLimit applies. Registry currently allocates the allowance in revision order rather than exact
cross-executor accrual order during overlapping refreshes. That MVP limitation is tracked in
GOL-848.
Memory and storage exports retain measured hard-limit overshoot instead of clipping it to capacity. Fractional byte-time remains exact across settlements and policy/month cutovers. Overshoot totals are diagnostics, not a second capacity debit. Billing eligibility is separate from measured usage. Registry limit requests carry the deployment's existing meter flags; disabled runtime dimensions skip usage reads and allowance calculations. All-off reads resolve per-agent policy without creating monthly billing revisions or baselines, and aggregate display does not create runtime billing state.
Registry replies carry delivery acknowledgement separately from monthly-policy availability. A
committed usage update can be acknowledged even when the next policy read fails. The account entry
marks enforcement unavailable without installing the fallback's mode, period or capacity. Admission,
guest checks and the existing monitor report infrastructure failure rather than quota exhaustion.
The entry retains the latest observed policy revision as a floor, so an older reply cannot restore
capacity. Unavailable entries refresh on the next existing batch tick; a valid policy restores
admission without resending acknowledged usage. All-off entries have no monthly availability gate.
Guest-entry, post-instantiation and epoch checks share worker/invocation.rs::resource_capacity_error.
Only typed resource exhaustion becomes durable suspension. Infrastructure errors retain their
AgentError and retry-policy classification; the epoch boundary preserves raw interruption for a
real suspension.
Fuel keeps the existing prepaid/refund meter. Production Wasmtime call hooks sample the physical
gauge at guest/host boundaries. FuelUsage keeps a separate metered gauge that decreases only
while that Store executes live guest code inside an actual guest-body scope. Physical fuel remains
visible in invocation diagnostics. Core instantiation, completed replay and snapshot loading do not
borrow or charge customer fuel. Snapshot saving keeps its existing live accounting.
A remaining prepaid reservation can retain its borrow
revision and month across a transition, with a default remaining batch of 10,000 fuel. Failed Registry
delivery can also lose captured usage; ambiguous additive updates are not blindly resent. Delivery
and customer-credit safety are tracked in
GOL-847.
Disabled meters omit their monthly accounting, caps, overage, and billing. Filesystem metering controls both storage meters but their amounts and usage remain separate. Per-agent memory and managed-XFS quota enforcement remain independent; unmanaged storage has no finite per-agent quota.
The single-binary configuration rejects enabled per-agent storage policy without managed XFS, independently of filesystem billing. Split-service executors reject a finite Registry storage limit on unaccounted storage instead of silently resolving it to Unlimited. Startup validates this before filesystem creation. A live limit update uses the same resolver and the existing seal/unload path on failure. Unlimited policy remains available on unmanaged storage; Registry's finite policy is a deployment requirement, not permission for an unsupported backend to omit enforcement.
A live invocation still holds its allocated linear memory and filesystem stock while waiting for host I/O. Its live allocations accrue byte-time during these permit-held waits even when no Wasm executes. The memory meter measures allocated linear-memory bytes, not the configured maximum. The parked guest itself burns no fuel.
Reconstruction keeps physical admission, the concurrent-agent permit, allocation observations,
per-agent limits and cleanup ownership. Materialization and replay do not start customer byte-time.
GuestResourceUsageGuard covers exported guest calls, native bodies and stream drains. Successful
PendingReplayToLive::finish_at activates usage only inside such a body. A primary replay tail outside
the body therefore does not bill final filesystem verification. Completed entity replay remains
unbillable; an incomplete entity activates its own memory after its local live admission succeeds,
even while the primary is still replaying. Its activation does not bill primary replay memory.
A filesystem-capable live body activates the generation's current storage stock. An incapable entity does not. That live stock remains billable through cleanup and permit release; separate blob stock accounting is unchanged. Entity activation shares the owner's memory-transition lock with window closure, so a late activation cannot resume a closed or dormant window.
The concurrent-agent permit bounds the billing window but does not itself make replay billable.
Permit release settles and pauses metering
before publishing reclaimable LoadedIdle or WarmRunnable state. This ends the billable window by
design; it does not mean every cached allocation is physically freed immediately. Released-permit
durable sleep, reclaimable cache time and unloaded time do not accrue memory or storage byte-time.
worker/monthly_limits/mod.rs attaches one monitor to an enabled, permit-held execution window. It checks
local capacity on a normal 30-second cadence and on applied Registry updates, settling owner-scoped
byte-time first and reading published fuel reservations without locking the Store. Actual changes
to billable memory also wake the monitor through the memory meter's rate watch. The monitor
subscribes before reading the rate and retains changes across its read/wait gap. Rate changes do
not publish a policy revision. Before each wait,
it settles that usage and wakes earlier when exact remaining monthly memory byte-nanoseconds divided
by the primary runtime's billable allocated linear-memory bytes predicts exhaustion sooner. The
horizon rounds up to a nanosecond; zero allocation, exhausted or inapplicable memory caps retain the
normal cadence, including rejected-stop retries. This owner-rate estimate is not a global account
prediction or a universal interruption deadline. It neither scans other
account owners nor changes Registry refresh cadence. All-off windows have no monthly monitor or
capacity subscription. Window closure invalidates its target and joins the monitor. During unload,
the filesystem drains before its final allocation read. The execution
window then settles and releases its permit before filesystem deletion, as in the existing unload
path. Failed deletion retains its filesystem reservation and explicit repair obligation, not the
closed billing window or execution permit. Repair verifies physical cleanup without reopening or
resettling the closed window. Cleanup failures still prevent startup on that Worker. Physical cleanup
receipts remain unfinished through failed deletion and complete only after verified repair.
Ordinary window close owns the final allocation sample and its existing deadline. There is no
separate frozen-allocation phase. Final-read retry and customer-favoring timeout behavior stay as
in main. A fault published during active-window enforcement remains failed health through close
and verified filesystem deletion; a transient final read alone does not create that retained fault.
Ordinary startup rollback after a window opens uses the same unload path. This does not cover
independent startup panic or cancellation: the filesystem and execution-window Drop fallbacks
have separate physical tasks, and the panic notification does not join them.
Initial idle settlement precedes startup readiness. Monitor panics, lost delivery and settlement
failures survive successful filesystem deletion as CleanupFailed; they
neither request immediate reconstruction nor acquire a replacement permit. The monitor treats a
closed lifecycle channel or lost send as an infrastructure error, retains that health, and requests
a cooperative Worker stop. Filesystem observation timeout or failure publishes retained window
health to that same monitor. Missing storage measurement is unavailable, not a zero sample or quota
exhaustion. The existing read remains owned and final allocation/drain work is still joined before
physical release. Known live memory continues accruing during that wait; unavailable storage intervals
do not fabricate measurements. A late successful read does not erase the window's failure.
If status work then panics because the actor has stopped, the invocation
loop retains its resident agent across the borrowed inner-loop failure and runs ordinary unload
without submitting more actor work. Tests in tests/api/interruption/lifecycle/health.rs cover initial
settlement, later successful invocations and pending silent TCP. Other panic paths still use the outer
panic boundary.
The monitor has no test-only panic selector or panic token on its target. Native health tests supply a panicking adapter through the existing weak owner-usage-flusher interface. Initial settlement, selected-result recording and already-published success need distinct proof boundaries; see idle health with distinct success boundaries. Cleanup, startup refusal and the original error remain strict without changing the monitor's await graph. The actor case holds the actual Start commit of a later live RPC after observing the successful allocation result and its real idle release; no idle-close control remains. Monthly monitoring uses the same ordinary Tokio timer in production and test builds.
Actor-loss tests fail a real positive GrowMemory append through the existing oplog fixture. Counter.allocate_memory supplies guest allocation; the TCP case uses its running invocation. The existing entity-fence log can establish stop-path entry, not physical drainage. Check the lost-actor error, physical deletion, retained failed health, startup refusal and original invocation records. Execution-lifetime tracing does not supply generation or permit-acquisition counts. Initial idle precedes guest execution, so a guest-growth fault cannot establish that initial boundary.
The settlement-loss selector is removed. It previously completed settlement successfully, then dropped the reply through Worker and window flags. The ordinary ObserverLost fallback remains. A colocated test faults the actual spawned close task through its existing MeteringClock dependency, requires the exact error and no successful settlement, and then requires the same real FIFO contender to acquire capacity. It adds no runtime selector or Worker interface. Missing acknowledgement alone does not prove that settlement failed or authorize settling again; this test's known clock fault occurs before settlement. The native fake reply-drop cells retire with their producer; the native health helper retains its other exact cleanup and startup assertions. This is not native settlement-task panic coverage. Earlier library, native, lint, production and browser results are historical; combined verification after instrumentation removal is pending. The post-success test uses the real RPC commit boundary described in the linked testing patterns.
A temporary Registry policy outage is an execution failure, not a disposal failure. The monitor requests the same cooperative stop and retains this error separately from cleanup health. After successful settlement and physical disposal, the ordinary unloaded-startup path reports the infrastructure error. A valid policy refresh allows the same cached durable Worker to restart through ordinary admission and reconstru
…(truncated)