# Substrate

> How AX depends on Agent Substrate (modules, address contract, mocks, version bumps, E2E pin). Load when touching internal/substrate, bumping Substrate, or triaging make test-e2e.

- Skill: `google/substrate` (Agent Skill)
- Install (CLI): `npx get-skill add google/substrate`
- Raw SKILL.md: https://api.skillmd.com/api/skills/google/substrate/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: Google (https://skillmd.com/u/google)
- Updated: 2026-11-16
- Page: https://skillmd.com/skills/google/substrate

---


# Agent Substrate in AX

AX schedules every `Task` as a sandboxed **actor** on
[Agent Substrate](https://github.com/agent-substrate/substrate). AX owns the
declarative layer (Task/Workspace/Model, reconciler, CLI); Substrate owns
isolation, checkpoint/restore, worker placement and in-cluster routing.
Facts below were verified against `main` on 2026-11-06 — re-check `go.mod`
and `hack/setup-e2e-kind.sh` if they look stale.

## 1. Module map

| Module | Pinned | Provides | Imported by |
|---|---|---|---|
| `go.ate.dev/substrate` | `v1.0.0` | Control API protos `api/ateapi/v1` (atespaces, actor templates, actors, tags, snapshot fidelity, suspend/resume, egress rules) | `internal/substrate`, `internal/substrate/substratetest`, `internal/server`, `internal/controller`, `internal/apivalidation`, `test/e2e`, `pkg/apis/v1alpha1` (its `.proto` files import `ateapi.proto`) |
| `github.com/agent-substrate/env` | `v0.0.11-0.20260912052224-4468a200b170` (pseudo-version) | Guest services served inside the sandbox (process exec + file system gRPC) and `proto/ateenv/v1alpha` | `internal/guest`, `internal/metadata` |

There is no `replace` directive for either module. Local hacking against a
Substrate checkout goes through `SUBSTRATE_DIR` in E2E (section 5).

## 2. The Control API wrapper: `internal/substrate.Client`

All Control API traffic goes through this package. It encapsulates:

- **Endpoint defaults** — `api.ate-system.svc.cluster.local:443`; authority
  and TLS `ServerName` default to `api.ate-system.svc` when the target
  contains that string.
- **Auth** — bearer token from `SUBSTRATE_TOKEN_FILE` (in-cluster: projected SA
  token with audience `api.ate-system.svc`, see `deploy/ax-server.yaml`), CA
  from `SUBSTRATE_CA_FILE`.
- **Retry** — `IsTransientError` + `callWithRetry` (`retryOnCodes`) for `Unavailable`-class errors.
- **Helpers** the reconcilers rely on — `EnsureAtespace`, `EnsureActorTemplate`,
  `EnsureActorTemplateWithImage`, `BuildActorTemplate`, `EnsureActor`,
  `EnsureActorFromTag`, `GetActorEgressPolicy`, `EnsureActorEgressPolicy`,
  `CreateTag`, `GetTag`, `ListTags`, `DeleteTag`, `SuspendActor`, `ResumeActor`,
  `DeleteActor`, `WorkerIP`, `ResourceLimits`.

### Where a new Control API call lives today

- `internal/substrate/client.go` is the only non-test file that calls
  `ateapipb.NewControlClient`; every existing RPC is a method on `*Client`,
  wrapped in `callWithRetry` (`codes.Unavailable` plus optional codes such as `codes.Aborted`).
- Each method has a table-driven case in `client_test.go` (`mockControlClient`).
- Callers in `internal/controller` exercise it through `mockControlServer`;
  `internal/server` and `cmd/ax` tests use `internal/substrate/substratetest`
  (section 4).

## 3. Address contract

| Value | Occurrences |
|---|---|
| `api.ate-system.svc.cluster.local:443` + authority `api.ate-system.svc` | `cmd/ax-server/main.go` (`--substrate-endpoint` / `--substrate-authority` flag defaults), `internal/substrate/client.go` (3×), `deploy/ax-server.yaml`, `test/e2e/manifests/ax-server.yaml`, `test/e2e/harness_test.go` |
| `atenet-router.ate-system.svc[.cluster.local]` | `internal/controller/reconciler.go` (`defaultRouterServerName`), `internal/tunnel/router_tls.go` (`routerServerName`), `cmd/ax/main.go` (ssh port-forward path), `internal/guest/client.go` (doc comment), `deploy/ax-server.yaml`, `test/e2e/manifests/ax-server.yaml`, `docs/networking.md` |
| namespace `ate-system` | all of the above + `test/e2e/harness_test.go` (`ateSystemNamespace`), `test/e2e/manifests/credential-policy.yaml` |
| pod labels `ate.dev/worker-pool` (worker pods) and `app: atenet-egress` (egress gateway) | the `ax-server` NetworkPolicy in `deploy/ax-server.yaml` and `test/e2e/manifests/ax-server.yaml`, which keeps both away from ax-server |
| `opentelemetry-collector.otel-system.svc:4317` (the collector Substrate's kind overlay deploys, `manifests/ate-install/kind/ate-otel-config.yaml` in Substrate) | `test/e2e/manifests/ax-server.yaml` (`ax-otel-config` ConfigMap). An unreachable address does not fail E2E; ax-server only logs OTLP export errors |

Why it matters: tasks never get a Service/Ingress. Every request to a task is
proxied by the **atenet router**, which picks the target from the
`ate-target-actor` header and resumes a suspended actor on demand
(`docs/networking.md`). `ax ssh` port-forwards to `svc/atenet-router` on 443
(mTLS) and falls back to 80.

## 4. Testing against Substrate

### Unit tests: in-process mock, no cluster

```go
type mockControlServer struct {
    ateapipb.UnimplementedControlServer
    // record calls / inject failures here
}
grpcServer := grpc.NewServer()
ateapipb.RegisterControlServer(grpcServer, mockSrv)
// serve on a listener, point substrate.NewClient at it
```

Canonical examples: `internal/controller/reconciler_test.go` (reconciler state
transitions; the largest test file in the repo). `internal/server/server_test.go`
and `cmd/ax/main_test.go` run a real `TaskReconciler` against
`internal/substrate/substratetest`, an in-memory fake of the same API (atespaces,
actor templates, actors, and tags).
`internal/substrate/client_test.go` uses a client-side `mockControlClient`
instead. No mocking library is in `go.mod`.

### Substrate behaviour the code already depends on

- Actor template images must be digest-pinned (`@sha256:…`); a tag-only
  `spec.profile.image` fails at ActorTemplate creation with reason
  `TemplateCreationFailed` (`docs/manifests.md`). `make images` writes
  `bin/ax-server.digest` / `bin/ax-task-runner.digest` for this reason.
- After `ResumeActor` the actor may be on a different worker, so AX never
  stores a worker IP: the reconciler uses the IP `ResumeActor` returns only for
  its readiness probe (`internal/controller/reconciler.go`), and `ax ssh`
  connects through `atenet-router`.
- Egress: an actor without an egress policy has no outbound access. For a
  task that binds a Gateway, `bindGateway` (`internal/controller/reconciler.go`)
  asks Substrate whether the actor has a policy. If it has none, `bindGateway`
  copies the Gateway's rules to the actor as its one policy (named `default`).
  An actor that has a policy keeps it, so the rules are copied once and never
  updated. `bindGateway` runs after the actor is suspended (`CreateTask`,
  `SuspendTask`) and before it is resumed (`ResumeTask`). Substrate validates
  the rules when they are copied. `bindGateway` fails with reason
  `GatewayBindingFailed` when Substrate refuses them, or when the actor has no
  policy and the Gateway cannot be loaded. `SuspendTask` and `ResumeTask` do
  not fail on the Gateway themselves (`fetchGateway` in
  `internal/server/server.go`), so a suspend always reaches `SuspendActor`. A
  task without a Gateway makes no egress policy call at all.
- `status.actorUid` is read live by `setActorStatus` in
  `internal/server/server.go` and cleared by `storedStatus` before writes.
  It changes when Substrate recreates the actor and is empty when the actor
  is missing or cannot be read. `ax describe task` shows it as `Actor UID`.
- Guest services (process + fs) are mounted by `internal/metadata/server.go`
  only when `task.Spec.Debug` is true; `ax ssh` (`cmd/ax/main.go`) refuses to
  connect otherwise.
- Trust bundle: `BuildActorTemplate` mounts `egress-mitm.ate.dev` and
  `system-roots.ate.dev` at `v1alpha1.EgressTrustBundlePath` for every task,
  and the runner points the CA variables at it (`runner/env.go`). A
  Substrate install that lacks either bundle blocks actor startup.
- Credential injection replaces a header only on requests that carry it. The
  runner supplies a placeholder only for git: `GIT_ASKPASS` is
  `/ax/git-askpass`, a link to the running program, and package `runner`
  answers from `init` when a program that imports it is started under that
  name (`runner/env.go`). It sets no API key variable, because tools choose a
  provider by which key is set: a task sets its own placeholders in
  `spec.env`. A Gateway's spec is never passed into the sandbox.
- Readiness: Substrate and the reconciler probe the runner on port 80
  (`/healthz`, `/readyz?check=workspace`, served by `internal/metadata`);
  `/readyz` is 503 until all workspaces are materialized (`docs/sandbox.md`).
- Snapshot storage: Actor templates created by AX leave `SnapshotConfig.StorageLocation`
  empty by default so Substrate uses its cluster-wide `--snapshot-location`. Specifying a custom
  snapshot bucket (`snapshotsBucket` in `BuildActorTemplate`) can only be used when the
  Substrate cluster has the `PerTemplateSnapshotLocation` preview feature gate enabled;
  otherwise Substrate rejects template creation with
  `actor_template.snapshot_config.storage_location: Forbidden`. Since v1.0.0 each path
  segment may contain only `[A-Za-z0-9_-]` and no segment may be `atespaces`; AX passes
  the value through unvalidated, so Substrate is the one that rejects it.

## 5. E2E and the Substrate version pin

`test/e2e` is compiled by `go test ./...` but **self-skips** unless
`AX_E2E=true` or `-args --e2e`. The real runs go through `make`:

| Target | Substrate version | When |
|---|---|---|
| `make test-e2e` | exactly `go list -m -f '{{.Version}}' go.ate.dev/substrate` | PR CI (skipped when only `paths-ignore` files change, e.g. docs) and the nightly schedule (06:00 UTC) in `e2e.yml` |
| `make test-e2e SUBSTRATE_REF=<release tag>` | override | local, or `workflow_dispatch` input `substrate_ref` in `e2e.yml` |
| `make e2e-setup` / `e2e-run` / `e2e-cleanup` | same as above | iterate without re-provisioning |

E2E never builds Substrate. `hack/setup-e2e-kind.sh` installs the release's
published images from `ghcr.io/agent-substrate/substrate/<image>:<tag>`
(`SUBSTRATE_IMAGE_REPO` to use a mirror) via `ATE_IMAGE_REPO`/`ATE_IMAGE_TAG`,
and pins the WorkerPool's worker image by digest. It exits early unless:

- `SUBSTRATE_REF` is a release tag, `vX.Y.Z` or `vX.Y.Z-rc.N`. Branches,
  commits, and Go pseudo-versions are rejected, and there is no run against
  Substrate `main`.
- The Substrate checkout is clean and at that tag. It still supplies the
  manifests and `ate-setup`, so they must match the images.
- That release's `ate-setup` supports `--image-repo`, and every image the
  install uses exists under the tag.

The install enables the `k8s.io` credential provider. For
`TestE2E_GatewayEgress` the script then:

- Applies `test/e2e/manifests/credential-policy.yaml`, which lets tasks in
  atespace `default` use Secrets labeled `ax.io/credential: "true"` in the AX
  namespace, and restarts the provider, which reads its policy only at startup.
- Runs Substrate's `hack/setup-e2e-egress-tls-kind.sh`, which makes the egress
  gateway trust Substrate's service-DNS CA.
- Deploys the echo server (`test/e2e/manifests/echo.yaml`), whose certificate
  comes from that CA.

Useful env (full list in `test/e2e/README.md`): `SUBSTRATE_DIR` (local
checkout; a detached worktree is created under `.tmp/` if HEAD ≠ ref),
`SUBSTRATE_REPO`, `E2E_ARGS='-test.run TestE2E_ModelCRUD'`.

## 6. Bumping Substrate

What the repo mechanically requires when `go.ate.dev/substrate` or
`github.com/agent-substrate/env` changes in `go.mod`:

```bash
go get go.ate.dev/substrate@vX.Y.Z            # and/or github.com/agent-substrate/env@<sha>
go mod tidy                                    # CI diffs go.mod/go.sum
make generate                                  # buf reads ateapi.proto from the new version
go build ./... && go test ./...                # unit tests compile against the new ateapipb
make test-e2e                                  # hack/setup-e2e-kind.sh installs the go.mod version
```

`go.ate.dev/substrate` must stay on a release tag: E2E installs that
release's published images, and a pseudo-version fails E2E setup.

`Gateway.spec.egress` is Substrate's `EgressRule`, so a bump can change what
users may write in a Gateway. Diff the egress messages in
`api/ateapi/v1/ateapi.proto` between the two versions. If the new version adds
a rule kind besides `http`, `https`, and `tlsPassthrough`, `ValidateGateway`
(`internal/apivalidation/gateway.go`) rejects rules of that kind until its
checks cover it. `docs/manifests.md` links to Substrate's egress credential
injection guide at the pinned version.

