Agent Substrate in AX
AX schedules every Task as a sandboxed actor on
Agent Substrate. AX owns the
declarative layer (Task/Workspace/Model, reconciler, CLI); Substrate owns
isolation, checkpoint/restore, worker placement and in-cluster routing.
Facts below were verified against main on 2026-11-06 — re-check go.mod
and hack/setup-e2e-kind.sh if they look stale.
1. Module map
| Module |
Pinned |
Provides |
Imported by |
go.ate.dev/substrate |
v1.0.0 |
Control API protos api/ateapi/v1 (atespaces, actor templates, actors, tags, snapshot fidelity, suspend/resume, egress rules) |
internal/substrate, internal/substrate/substratetest, internal/server, internal/controller, internal/apivalidation, test/e2e, pkg/apis/v1alpha1 (its .proto files import ateapi.proto) |
github.com/agent-substrate/env |
v0.0.11-0.20260912052224-4468a200b170 (pseudo-version) |
Guest services served inside the sandbox (process exec + file system gRPC) and proto/ateenv/v1alpha |
internal/guest, internal/metadata |
There is no replace directive for either module. Local hacking against a
Substrate checkout goes through SUBSTRATE_DIR in E2E (section 5).
2. The Control API wrapper: internal/substrate.Client
All Control API traffic goes through this package. It encapsulates:
- Endpoint defaults —
api.ate-system.svc.cluster.local:443; authority
and TLS ServerName default to api.ate-system.svc when the target
contains that string.
- Auth — bearer token from
SUBSTRATE_TOKEN_FILE (in-cluster: projected SA
token with audience api.ate-system.svc, see deploy/ax-server.yaml), CA
from SUBSTRATE_CA_FILE.
- Retry —
IsTransientError + callWithRetry (retryOnCodes) for Unavailable-class errors.
- Helpers the reconcilers rely on —
EnsureAtespace, EnsureActorTemplate,
EnsureActorTemplateWithImage, BuildActorTemplate, EnsureActor,
EnsureActorFromTag, GetActorEgressPolicy, EnsureActorEgressPolicy,
CreateTag, GetTag, ListTags, DeleteTag, SuspendActor, ResumeActor,
DeleteActor, WorkerIP, ResourceLimits.
Where a new Control API call lives today
internal/substrate/client.go is the only non-test file that calls
ateapipb.NewControlClient; every existing RPC is a method on *Client,
wrapped in callWithRetry (codes.Unavailable plus optional codes such as codes.Aborted).
- Each method has a table-driven case in
client_test.go (mockControlClient).
- Callers in
internal/controller exercise it through mockControlServer;
internal/server and cmd/ax tests use internal/substrate/substratetest
(section 4).
3. Address contract
| Value |
Occurrences |
api.ate-system.svc.cluster.local:443 + authority api.ate-system.svc |
cmd/ax-server/main.go (--substrate-endpoint / --substrate-authority flag defaults), internal/substrate/client.go (3×), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, test/e2e/harness_test.go |
atenet-router.ate-system.svc[.cluster.local] |
internal/controller/reconciler.go (defaultRouterServerName), internal/tunnel/router_tls.go (routerServerName), cmd/ax/main.go (ssh port-forward path), internal/guest/client.go (doc comment), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, docs/networking.md |
namespace ate-system |
all of the above + test/e2e/harness_test.go (ateSystemNamespace), test/e2e/manifests/credential-policy.yaml |
pod labels ate.dev/worker-pool (worker pods) and app: atenet-egress (egress gateway) |
the ax-server NetworkPolicy in deploy/ax-server.yaml and test/e2e/manifests/ax-server.yaml, which keeps both away from ax-server |
opentelemetry-collector.otel-system.svc:4317 (the collector Substrate's kind overlay deploys, manifests/ate-install/kind/ate-otel-config.yaml in Substrate) |
test/e2e/manifests/ax-server.yaml (ax-otel-config ConfigMap). An unreachable address does not fail E2E; ax-server only logs OTLP export errors |
Why it matters: tasks never get a Service/Ingress. Every request to a task is
proxied by the atenet router, which picks the target from the
ate-target-actor header and resumes a suspended actor on demand
(docs/networking.md). ax ssh port-forwards to svc/atenet-router on 443
(mTLS) and falls back to 80.
4. Testing against Substrate
Unit tests: in-process mock, no cluster
type mockControlServer struct {
ateapipb.UnimplementedControlServer
// record calls / inject failures here
}
grpcServer := grpc.NewServer()
ateapipb.RegisterControlServer(grpcServer, mockSrv)
// serve on a listener, point substrate.NewClient at it
Canonical examples: internal/controller/reconciler_test.go (reconciler state
transitions; the largest test file in the repo). internal/server/server_test.go
and cmd/ax/main_test.go run a real TaskReconciler against
internal/substrate/substratetest, an in-memory fake of the same API (atespaces,
actor templates, actors, and tags).
internal/substrate/client_test.go uses a client-side mockControlClient
instead. No mocking library is in go.mod.
Substrate behaviour the code already depends on
- Actor template images must be digest-pinned (
@sha256:…); a tag-only
spec.profile.image fails at ActorTemplate creation with reason
TemplateCreationFailed (docs/manifests.md). make images writes
bin/ax-server.digest / bin/ax-task-runner.digest for this reason.
- After
ResumeActor the actor may be on a different worker, so AX never
stores a worker IP: the reconciler uses the IP ResumeActor returns only for
its readiness probe (internal/controller/reconciler.go), and ax ssh
connects through atenet-router.
- Egress: an actor without an egress policy has no outbound access. For a
task that binds a Gateway,
bindGateway (internal/controller/reconciler.go)
asks Substrate whether the actor has a policy. If it has none, bindGateway
copies the Gateway's rules to the actor as its one policy (named default).
An actor that has a policy keeps it, so the rules are copied once and never
updated. bindGateway runs after the actor is suspended (CreateTask,
SuspendTask) and before it is resumed (ResumeTask). Substrate validates
the rules when they are copied. bindGateway fails with reason
GatewayBindingFailed when Substrate refuses them, or when the actor has no
policy and the Gateway cannot be loaded. SuspendTask and ResumeTask do
not fail on the Gateway themselves (fetchGateway in
internal/server/server.go), so a suspend always reaches SuspendActor. A
task without a Gateway makes no egress policy call at all.
status.actorUid is read live by setActorStatus in
internal/server/server.go and cleared by storedStatus before writes.
It changes when Substrate recreates the actor and is empty when the actor
is missing or cannot be read. ax describe task shows it as Actor UID.
- Guest services (process + fs) are mounted by
internal/metadata/server.go
only when task.Spec.Debug is true; ax ssh (cmd/ax/main.go) refuses to
connect otherwise.
- Trust bundle:
BuildActorTemplate mounts egress-mitm.ate.dev and
system-roots.ate.dev at v1alpha1.EgressTrustBundlePath for every task,
and the runner points the CA variables at it (runner/env.go). A
Substrate install that lacks either bundle blocks actor startup.
- Credential injection replaces a header only on requests that carry it. The
runner supplies a placeholder only for git:
GIT_ASKPASS is
/ax/git-askpass, a link to the running program, and package runner
answers from init when a program that imports it is started under that
name (runner/env.go). It sets no API key variable, because tools choose a
provider by which key is set: a task sets its own placeholders in
spec.env. A Gateway's spec is never passed into the sandbox.
- Readiness: Substrate and the reconciler probe the runner on port 80
(
/healthz, /readyz?check=workspace, served by internal/metadata);
/readyz is 503 until all workspaces are materialized (docs/sandbox.md).
- Snapshot storage: Actor templates created by AX leave
SnapshotConfig.StorageLocation
empty by default so Substrate uses its cluster-wide --snapshot-location. Specifying a custom
snapshot bucket (snapshotsBucket in BuildActorTemplate) can only be used when the
Substrate cluster has the PerTemplateSnapshotLocation preview feature gate enabled;
otherwise Substrate rejects template creation with
actor_template.snapshot_config.storage_location: Forbidden. Since v1.0.0 each path
segment may contain only [A-Za-z0-9_-] and no segment may be atespaces; AX passes
the value through unvalidated, so Substrate is the one that rejects it.
5. E2E and the Substrate version pin
test/e2e is compiled by go test ./... but self-skips unless
AX_E2E=true or -args --e2e. The real runs go through make:
| Target |
Substrate version |
When |
make test-e2e |
exactly go list -m -f '{{.Version}}' go.ate.dev/substrate |
PR CI (skipped when only paths-ignore files change, e.g. docs) and the nightly schedule (06:00 UTC) in e2e.yml |
make test-e2e SUBSTRATE_REF=<release tag> |
override |
local, or workflow_dispatch input substrate_ref in e2e.yml |
make e2e-setup / e2e-run / e2e-cleanup |
same as above |
iterate without re-provisioning |
E2E never builds Substrate. hack/setup-e2e-kind.sh installs the release's
published images from ghcr.io/agent-substrate/substrate/<image>:<tag>
(SUBSTRATE_IMAGE_REPO to use a mirror) via ATE_IMAGE_REPO/ATE_IMAGE_TAG,
and pins the WorkerPool's worker image by digest. It exits early unless:
SUBSTRATE_REF is a release tag, vX.Y.Z or vX.Y.Z-rc.N. Branches,
commits, and Go pseudo-versions are rejected, and there is no run against
Substrate main.
- The Substrate checkout is clean and at that tag. It still supplies the
manifests and
ate-setup, so they must match the images.
- That release's
ate-setup supports --image-repo, and every image the
install uses exists under the tag.
The install enables the k8s.io credential provider. For
TestE2E_GatewayEgress the script then:
- Applies
test/e2e/manifests/credential-policy.yaml, which lets tasks in
atespace default use Secrets labeled ax.io/credential: "true" in the AX
namespace, and restarts the provider, which reads its policy only at startup.
- Runs Substrate's
hack/setup-e2e-egress-tls-kind.sh, which makes the egress
gateway trust Substrate's service-DNS CA.
- Deploys the echo server (
test/e2e/manifests/echo.yaml), whose certificate
comes from that CA.
Useful env (full list in test/e2e/README.md): SUBSTRATE_DIR (local
checkout; a detached worktree is created under .tmp/ if HEAD ≠ ref),
SUBSTRATE_REPO, E2E_ARGS='-test.run TestE2E_ModelCRUD'.
6. Bumping Substrate
What the repo mechanically requires when go.ate.dev/substrate or
github.com/agent-substrate/env changes in go.mod:
go get go.ate.dev/substrate@vX.Y.Z # and/or github.com/agent-substrate/env@<sha>
go mod tidy # CI diffs go.mod/go.sum
make generate # buf reads ateapi.proto from the new version
go build ./... && go test ./... # unit tests compile against the new ateapipb
make test-e2e # hack/setup-e2e-kind.sh installs the go.mod version
go.ate.dev/substrate must stay on a release tag: E2E installs that
release's published images, and a pseudo-version fails E2E setup.
Gateway.spec.egress is Substrate's EgressRule, so a bump can change what
users may write in a Gateway. Diff the egress messages in
api/ateapi/v1/ateapi.proto between the two versions. If the new version adds
a rule kind besides http, https, and tlsPassthrough, ValidateGateway
(internal/apivalidation/gateway.go) rejects rules of that kind until its
checks cover it. docs/manifests.md links to Substrate's egress credential
injection guide at the pinned version.
1---2name: substrate3description: >4 How AX depends on Agent Substrate (modules, address contract, mocks, version bumps, E2E pin).5 Load when touching internal/substrate, bumping Substrate, or triaging make test-e2e.6metadata:7 source: https://skillmd.com/skills/google/substrate8---910# Agent Substrate in AX1112AX schedules every `Task` as a sandboxed **actor** on13[Agent Substrate](https://github.com/agent-substrate/substrate). AX owns the14declarative layer (Task/Workspace/Model, reconciler, CLI); Substrate owns15isolation, checkpoint/restore, worker placement and in-cluster routing.16Facts below were verified against `main` on 2026-11-06 — re-check `go.mod`17and `hack/setup-e2e-kind.sh` if they look stale.1819## 1. Module map2021| Module | Pinned | Provides | Imported by |22|---|---|---|---|23| `go.ate.dev/substrate` | `v1.0.0` | Control API protos `api/ateapi/v1` (atespaces, actor templates, actors, tags, snapshot fidelity, suspend/resume, egress rules) | `internal/substrate`, `internal/substrate/substratetest`, `internal/server`, `internal/controller`, `internal/apivalidation`, `test/e2e`, `pkg/apis/v1alpha1` (its `.proto` files import `ateapi.proto`) |24| `github.com/agent-substrate/env` | `v0.0.11-0.20260912052224-4468a200b170` (pseudo-version) | Guest services served inside the sandbox (process exec + file system gRPC) and `proto/ateenv/v1alpha` | `internal/guest`, `internal/metadata` |2526There is no `replace` directive for either module. Local hacking against a27Substrate checkout goes through `SUBSTRATE_DIR` in E2E (section 5).2829## 2. The Control API wrapper: `internal/substrate.Client`3031All Control API traffic goes through this package. It encapsulates:3233- **Endpoint defaults** — `api.ate-system.svc.cluster.local:443`; authority34 and TLS `ServerName` default to `api.ate-system.svc` when the target35 contains that string.36- **Auth** — bearer token from `SUBSTRATE_TOKEN_FILE` (in-cluster: projected SA37 token with audience `api.ate-system.svc`, see `deploy/ax-server.yaml`), CA38 from `SUBSTRATE_CA_FILE`.39- **Retry** — `IsTransientError` + `callWithRetry` (`retryOnCodes`) for `Unavailable`-class errors.40- **Helpers** the reconcilers rely on — `EnsureAtespace`, `EnsureActorTemplate`,41 `EnsureActorTemplateWithImage`, `BuildActorTemplate`, `EnsureActor`,42 `EnsureActorFromTag`, `GetActorEgressPolicy`, `EnsureActorEgressPolicy`,43 `CreateTag`, `GetTag`, `ListTags`, `DeleteTag`, `SuspendActor`, `ResumeActor`,44 `DeleteActor`, `WorkerIP`, `ResourceLimits`.4546### Where a new Control API call lives today4748- `internal/substrate/client.go` is the only non-test file that calls49 `ateapipb.NewControlClient`; every existing RPC is a method on `*Client`,50 wrapped in `callWithRetry` (`codes.Unavailable` plus optional codes such as `codes.Aborted`).51- Each method has a table-driven case in `client_test.go` (`mockControlClient`).52- Callers in `internal/controller` exercise it through `mockControlServer`;53 `internal/server` and `cmd/ax` tests use `internal/substrate/substratetest`54 (section 4).5556## 3. Address contract5758| Value | Occurrences |59|---|---|60| `api.ate-system.svc.cluster.local:443` + authority `api.ate-system.svc` | `cmd/ax-server/main.go` (`--substrate-endpoint` / `--substrate-authority` flag defaults), `internal/substrate/client.go` (3×), `deploy/ax-server.yaml`, `test/e2e/manifests/ax-server.yaml`, `test/e2e/harness_test.go` |61| `atenet-router.ate-system.svc[.cluster.local]` | `internal/controller/reconciler.go` (`defaultRouterServerName`), `internal/tunnel/router_tls.go` (`routerServerName`), `cmd/ax/main.go` (ssh port-forward path), `internal/guest/client.go` (doc comment), `deploy/ax-server.yaml`, `test/e2e/manifests/ax-server.yaml`, `docs/networking.md` |62| namespace `ate-system` | all of the above + `test/e2e/harness_test.go` (`ateSystemNamespace`), `test/e2e/manifests/credential-policy.yaml` |63| pod labels `ate.dev/worker-pool` (worker pods) and `app: atenet-egress` (egress gateway) | the `ax-server` NetworkPolicy in `deploy/ax-server.yaml` and `test/e2e/manifests/ax-server.yaml`, which keeps both away from ax-server |64| `opentelemetry-collector.otel-system.svc:4317` (the collector Substrate's kind overlay deploys, `manifests/ate-install/kind/ate-otel-config.yaml` in Substrate) | `test/e2e/manifests/ax-server.yaml` (`ax-otel-config` ConfigMap). An unreachable address does not fail E2E; ax-server only logs OTLP export errors |6566Why it matters: tasks never get a Service/Ingress. Every request to a task is67proxied by the **atenet router**, which picks the target from the68`ate-target-actor` header and resumes a suspended actor on demand69(`docs/networking.md`). `ax ssh` port-forwards to `svc/atenet-router` on 44370(mTLS) and falls back to 80.7172## 4. Testing against Substrate7374### Unit tests: in-process mock, no cluster7576```go77type mockControlServer struct {78 ateapipb.UnimplementedControlServer79 // record calls / inject failures here80}81grpcServer := grpc.NewServer()82ateapipb.RegisterControlServer(grpcServer, mockSrv)83// serve on a listener, point substrate.NewClient at it84```8586Canonical examples: `internal/controller/reconciler_test.go` (reconciler state87transitions; the largest test file in the repo). `internal/server/server_test.go`88and `cmd/ax/main_test.go` run a real `TaskReconciler` against89`internal/substrate/substratetest`, an in-memory fake of the same API (atespaces,90actor templates, actors, and tags).91`internal/substrate/client_test.go` uses a client-side `mockControlClient`92instead. No mocking library is in `go.mod`.9394### Substrate behaviour the code already depends on9596- Actor template images must be digest-pinned (`@sha256:…`); a tag-only97 `spec.profile.image` fails at ActorTemplate creation with reason98 `TemplateCreationFailed` (`docs/manifests.md`). `make images` writes99 `bin/ax-server.digest` / `bin/ax-task-runner.digest` for this reason.100- After `ResumeActor` the actor may be on a different worker, so AX never101 stores a worker IP: the reconciler uses the IP `ResumeActor` returns only for102 its readiness probe (`internal/controller/reconciler.go`), and `ax ssh`103 connects through `atenet-router`.104- Egress: an actor without an egress policy has no outbound access. For a105 task that binds a Gateway, `bindGateway` (`internal/controller/reconciler.go`)106 asks Substrate whether the actor has a policy. If it has none, `bindGateway`107 copies the Gateway's rules to the actor as its one policy (named `default`).108 An actor that has a policy keeps it, so the rules are copied once and never109 updated. `bindGateway` runs after the actor is suspended (`CreateTask`,110 `SuspendTask`) and before it is resumed (`ResumeTask`). Substrate validates111 the rules when they are copied. `bindGateway` fails with reason112 `GatewayBindingFailed` when Substrate refuses them, or when the actor has no113 policy and the Gateway cannot be loaded. `SuspendTask` and `ResumeTask` do114 not fail on the Gateway themselves (`fetchGateway` in115 `internal/server/server.go`), so a suspend always reaches `SuspendActor`. A116 task without a Gateway makes no egress policy call at all.117- `status.actorUid` is read live by `setActorStatus` in118 `internal/server/server.go` and cleared by `storedStatus` before writes.119 It changes when Substrate recreates the actor and is empty when the actor120 is missing or cannot be read. `ax describe task` shows it as `Actor UID`.121- Guest services (process + fs) are mounted by `internal/metadata/server.go`122 only when `task.Spec.Debug` is true; `ax ssh` (`cmd/ax/main.go`) refuses to123 connect otherwise.124- Trust bundle: `BuildActorTemplate` mounts `egress-mitm.ate.dev` and125 `system-roots.ate.dev` at `v1alpha1.EgressTrustBundlePath` for every task,126 and the runner points the CA variables at it (`runner/env.go`). A127 Substrate install that lacks either bundle blocks actor startup.128- Credential injection replaces a header only on requests that carry it. The129 runner supplies a placeholder only for git: `GIT_ASKPASS` is130 `/ax/git-askpass`, a link to the running program, and package `runner`131 answers from `init` when a program that imports it is started under that132 name (`runner/env.go`). It sets no API key variable, because tools choose a133 provider by which key is set: a task sets its own placeholders in134 `spec.env`. A Gateway's spec is never passed into the sandbox.135- Readiness: Substrate and the reconciler probe the runner on port 80136 (`/healthz`, `/readyz?check=workspace`, served by `internal/metadata`);137 `/readyz` is 503 until all workspaces are materialized (`docs/sandbox.md`).138- Snapshot storage: Actor templates created by AX leave `SnapshotConfig.StorageLocation`139 empty by default so Substrate uses its cluster-wide `--snapshot-location`. Specifying a custom140 snapshot bucket (`snapshotsBucket` in `BuildActorTemplate`) can only be used when the141 Substrate cluster has the `PerTemplateSnapshotLocation` preview feature gate enabled;142 otherwise Substrate rejects template creation with143 `actor_template.snapshot_config.storage_location: Forbidden`. Since v1.0.0 each path144 segment may contain only `[A-Za-z0-9_-]` and no segment may be `atespaces`; AX passes145 the value through unvalidated, so Substrate is the one that rejects it.146147## 5. E2E and the Substrate version pin148149`test/e2e` is compiled by `go test ./...` but **self-skips** unless150`AX_E2E=true` or `-args --e2e`. The real runs go through `make`:151152| Target | Substrate version | When |153|---|---|---|154| `make test-e2e` | exactly `go list -m -f '{{.Version}}' go.ate.dev/substrate` | PR CI (skipped when only `paths-ignore` files change, e.g. docs) and the nightly schedule (06:00 UTC) in `e2e.yml` |155| `make test-e2e SUBSTRATE_REF=<release tag>` | override | local, or `workflow_dispatch` input `substrate_ref` in `e2e.yml` |156| `make e2e-setup` / `e2e-run` / `e2e-cleanup` | same as above | iterate without re-provisioning |157158E2E never builds Substrate. `hack/setup-e2e-kind.sh` installs the release's159published images from `ghcr.io/agent-substrate/substrate/<image>:<tag>`160(`SUBSTRATE_IMAGE_REPO` to use a mirror) via `ATE_IMAGE_REPO`/`ATE_IMAGE_TAG`,161and pins the WorkerPool's worker image by digest. It exits early unless:162163- `SUBSTRATE_REF` is a release tag, `vX.Y.Z` or `vX.Y.Z-rc.N`. Branches,164 commits, and Go pseudo-versions are rejected, and there is no run against165 Substrate `main`.166- The Substrate checkout is clean and at that tag. It still supplies the167 manifests and `ate-setup`, so they must match the images.168- That release's `ate-setup` supports `--image-repo`, and every image the169 install uses exists under the tag.170171The install enables the `k8s.io` credential provider. For172`TestE2E_GatewayEgress` the script then:173174- Applies `test/e2e/manifests/credential-policy.yaml`, which lets tasks in175 atespace `default` use Secrets labeled `ax.io/credential: "true"` in the AX176 namespace, and restarts the provider, which reads its policy only at startup.177- Runs Substrate's `hack/setup-e2e-egress-tls-kind.sh`, which makes the egress178 gateway trust Substrate's service-DNS CA.179- Deploys the echo server (`test/e2e/manifests/echo.yaml`), whose certificate180 comes from that CA.181182Useful env (full list in `test/e2e/README.md`): `SUBSTRATE_DIR` (local183checkout; a detached worktree is created under `.tmp/` if HEAD ≠ ref),184`SUBSTRATE_REPO`, `E2E_ARGS='-test.run TestE2E_ModelCRUD'`.185186## 6. Bumping Substrate187188What the repo mechanically requires when `go.ate.dev/substrate` or189`github.com/agent-substrate/env` changes in `go.mod`:190191```bash192go get go.ate.dev/substrate@vX.Y.Z # and/or github.com/agent-substrate/env@<sha>193go mod tidy # CI diffs go.mod/go.sum194make generate # buf reads ateapi.proto from the new version195go build ./... && go test ./... # unit tests compile against the new ateapipb196make test-e2e # hack/setup-e2e-kind.sh installs the go.mod version197```198199`go.ate.dev/substrate` must stay on a release tag: E2E installs that200release's published images, and a pseudo-version fails E2E setup.201202`Gateway.spec.egress` is Substrate's `EgressRule`, so a bump can change what203users may write in a Gateway. Diff the egress messages in204`api/ateapi/v1/ateapi.proto` between the two versions. If the new version adds205a rule kind besides `http`, `https`, and `tlsPassthrough`, `ValidateGateway`206(`internal/apivalidation/gateway.go`) rejects rules of that kind until its207checks cover it. `docs/manifests.md` links to Substrate's egress credential208injection guide at the pinned version.