# Agent Substrate Overview

> Reference for the alpha agent-substrate/substrate project as an alternative ultra-scale control plane that shares snapshot and runtime primitives with kubernetes-sigs/agent-sandbox. Use when comparing agent-sandbox vs Substrate, evaluating the WorkerPool + ActorTemplate multiplexing model, reviewing the kubectl-ate CLI, or deciding which control plane fits a workload that needs millions of sub-second agent tool calls.

- Skill: `vinnie357/agent-substrate-overview` (Agent Skill)
- Install (CLI): `npx skillmds add vinnie357/agent-substrate-overview`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vinnie357/agent-substrate-overview/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: MIT
- Author: vinnie357 (https://skillmd.com/u/vinnie357)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/vinnie357/agent-substrate-overview

---


# agent-substrate-overview

Agent Substrate is an Apache-2.0 Kubernetes runtime for agent-like workloads that multiplexes many stateful actors onto fewer worker pods, preserving in-memory + disk state via gVisor checkpoint/restore. It is **explicitly alpha** (v0.0.0, README quote: "VERY early development… APIs are almost guaranteed to change"). This skill is a short reference so you can evaluate it against `k8s-agent-sandbox`, not an operational guide.

If the operational decision is already "use Substrate," read the upstream README directly — this skill is intentionally not a step-by-step install guide because Substrate is at v0.0.0 with API churn between commits, so pinning a recipe here would go stale.

## What Substrate is

From the upstream README: "Agent substrate is a system built on top of Kubernetes which manages agent-like workloads to achieve higher scale and efficiency than Kubernetes alone can offer, with lower latency."

Architectural diff vs `kubernetes-sigs/agent-sandbox`:

| | `agent-sandbox` (kubernetes-sigs) | `agent-substrate` |
|---|---|---|
| Maturity | v0.4.6 (released) | v0.0.0 (alpha) |
| Control plane | k8s controller + CRDs only | k8s controller + dedicated gRPC `ateapi` |
| Workload model | One sandbox per claim | Many actors multiplexed onto fewer pods |
| Suspend/resume | Pod snapshots (GKE feature) | gVisor `runsc checkpoint/restore` first-class |
| CLI | `kubectl` + Python SDK | `kubectl-ate` plugin |
| Use case | per-session ephemeral agents | high-density actor swarms |

Per Google's 2026-05-20 blog: "Substrate is the open-source ultra-scale tier sharing Sandbox's core secure runtime and snapshotting capabilities with a minimal control plane designed to bypass some of the limitations of Kubernetes for millions of sub-second tool calls."

## Components

- `ateapi` — gRPC control plane.
- `atelet` — node-level DaemonSet supervising workers and snapshots.
- `atecontroller` — k8s controller reconciling Substrate CRDs.
- `atenet` — networking controller: DNS, Envoy routing, proxy sidecars.
- `ateom-gvisor` — interior helper running `runsc checkpoint/restore`.
- `podcertcontroller` — Pod Certificate signer polyfill.
- `kubectl-ate` — operator CLI plugin.

External deps: gVisor, Valkey (Redis-compatible), RustFS.

## CRDs

- `WorkerPool` — N pods that will host actors. Configures pod sizing and oversubscription.
- `ActorTemplate` — describes a workload container image + boot/snapshot config; many actors instantiate from one template.

`SessionIdentity` (not a CRD; a service exposed by `ateapi`) exchanges ephemeral kubelet credentials for stable JWT/cert identities that persist across worker migrations.

## Claude Code on Substrate

Substrate's `demos/claude-code-multiplex/` ships a working example:

- A `WorkerPool` of 2 pods is oversubscribed by 3 `ActorTemplate`s (`luna`, `mars`, `orion`).
- Each agent is a container built from `workload/` (Dockerfile + Python wrapper around Claude Code) and referenced by `sha256` digest in `claude-code-multiplex.yaml.tmpl`.
- A Go dashboard in `ui/` calls the `ateapi` gRPC + Kubernetes API.

## Example: WorkerPool + ActorTemplate field semantics

Trimmed from `demos/claude-code-multiplex/claude-code-multiplex.yaml.tmpl` upstream (Apache-2.0) — the `WorkerPool` plus one of its three `ActorTemplate`s (`mars` and `orion` are structurally identical to `luna`, differing only in `metadata.name` and the `ACTOR_NAME`/`TASK` env values):

```yaml
# 2 worker replicas for 3 actors — the multiplex pressure that makes the
# substrate suspend/resume behavior visible.
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: claude-workerpool
  namespace: claude-multiplex-demo
  labels:
    workload: claude-multiplex
spec:
  replicas: 2
  ateomImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor

---

apiVersion: ate.dev/v1alpha1
kind: ActorTemplate
metadata:
  name: agent-luna
  namespace: claude-multiplex-demo
spec:
  pauseImage: "registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4"
  containers:
  - name: claude
    image: ${WORKLOAD_IMAGE}
    env:
    - name: ACTOR_NAME
      value: "luna"
    - name: TASK
      value: "Tell me one short, surprising fact about the Moon. One sentence."
    - name: ANTHROPIC_API_KEY
      valueFrom:
        secretKeyRef:
          name: anthropic-api-key
          key: api-key
  workerSelector:
    matchLabels:
      workload: claude-multiplex
  snapshotsConfig:
    location: gs://${BUCKET_NAME}/claude-multiplex-demo/
```

Field semantics not obvious from the names alone:

- `WorkerPool.spec.replicas` — pod count, independent of how many `ActorTemplate`s target it. Here, 2 pods host 3 actors, so Substrate must suspend at least one actor at any moment.
- `WorkerPool.spec.ateomImage` — the `ateom-gvisor` interior helper image each pod runs to perform checkpoint/restore; not the workload image itself.
- `ActorTemplate.spec.workerSelector.matchLabels` — restricts which `WorkerPool`s this template's actors may schedule onto, by matching pool `metadata.labels`. Typed `*metav1.LabelSelector` (the Deployment/NetworkPolicy idiom, not a Service's bare `spec.selector` map) and optional — upstream docs it as a gate: "if nil, all pools are eligible," and it can only narrow the eligible set, never expand it.
- `ActorTemplate.spec.snapshotsConfig.location` — where suspended-actor state is persisted (an object-storage URI here) so a later resume can restore it, potentially onto a different pod.
- `ActorTemplate.spec.pauseImage` — the sandbox/pause container each actor's pod slot boots from; unrelated to the workload container image.

Substrate is **not** an MCP server — it's a workload host. Claude Code packaged as an OCI container becomes an actor like any other Substrate workload.

## Install summary (do not paste verbatim — read upstream first)

The upstream uses shell scripts under `hack/`, not Helm or Kustomize. Examples:

- Kind: `hack/create-kind-cluster.sh`, `hack/install-ate-kind.sh --deploy-ate-system`
- GKE: `go run ./tools/setup-gcp --all`, `hack/install-ate.sh --deploy-ate-system`

The exact scripts change frequently in alpha. Read the README at https://github.com/agent-substrate/substrate before running any of them.

## Which to pick

- **Use `kubernetes-sigs/agent-sandbox`** for: per-session Claude Code pods on a laptop or GKE; production workloads now; integration with the broader k8s ecosystem (NetworkPolicy, StorageClass, etc.).
- **Use `agent-substrate`** for: ultra-high-density actor workloads where standard k8s scheduling is the bottleneck; experimentation with checkpoint/restore migration; research environments where alpha APIs are acceptable.

For this plugin's Claude Code-on-sandbox use case (one operator, one session at a time, microVM isolation): **stick with `k8s-agent-sandbox`**. Substrate's multiplexing is overkill and its alpha status means upgrade churn.

## See also

- `k8s-agent-sandbox` — the primary path this plugin recommends.
- Upstream Substrate: https://github.com/agent-substrate/substrate
- 2026-05-20 GCP blog: https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate

## Anti-fabrication

<!-- vale Flavored.Superlatives = NO -->
- Substrate is at v0.0.0; do not claim it is production-ready.
<!-- vale Flavored.Superlatives = YES -->
- The `kubectl-ate` plugin and `ateapi` are different things from agent-sandbox's `kubectl` + Python SDK — do not paste agent-sandbox CRDs into a Substrate cluster or vice versa.
- The "shares snapshot primitives" relationship is documented in Google's blog post; the actual code-level overlap is not enumerated upstream. Don't claim deeper code-level equivalence than that.
- Substrate's install scripts change frequently. Read the current README before invoking them.