# Helm Dev Environment

> Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local, high availability, HA.

- Skill: `nvidia/helm-dev-environment` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nvidia/helm-dev-environment`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/helm-dev-environment/raw
- Safety review: pending (external: skill-scanner PASS, skillspector WARNING)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/nvidia/helm-dev-environment

---


# Helm Dev Environment

Set up, run, and tear down the local Kubernetes development environment for OpenShell.
The stack is: **k3d** (Docker-backed k3s) for the cluster, **Skaffold** for image builds and Helm deploys, and the **OpenShell Helm chart** (`deploy/helm/openshell/`).

---

## Prerequisites

- Docker Desktop (macOS) or Docker Engine (Linux) running
- `mise install` completed (provides `k3d`, `kubectl`, `skaffold`, `helm`)

---

## Startup

### 1. Create the cluster

```bash
mise run helm:k3s:create
```

Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file.
When the named cluster already exists, the task starts any stopped containers and refreshes
same-named kubeconfig entries so a recreated load balancer's current API port takes effect.
Also applies the upstream agent-sandbox CRDs/controller (pinned via `AGENT_SANDBOX_VERSION`
in `tasks/scripts/helm-k3s-local.sh`, fetched from `github.com/kubernetes-sigs/agent-sandbox`
releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace
collector and UI in the `observability` namespace,
and preloads the default community sandbox image into k3d so the first sandbox create
does not wait on a large registry pull. Traefik is disabled at cluster creation time.

**Multi-worktree support:** the cluster name is derived from the last component of the
current git branch (e.g. branch `kube-support/local-dev/tmutch` → cluster
`openshell-dev-tmutch`). Each worktree therefore gets its own isolated cluster and its
own `kubeconfig` file. Override with `HELM_K3S_CLUSTER_NAME` to force a specific name
or share one cluster across worktrees.

Port mappings created at cluster time (cannot be changed without recreating):

| Host port | Target | Used by |
|-----------|--------|---------|
| `8080` | Port `80` via k3d load balancer | Envoy Gateway LoadBalancer service (`values-gateway.yaml`) |

Override with env vars before running `helm:k3s:create`:
- `HELM_K3S_LB_HOST_PORT` (default: `8080`)
- `HELM_K3S_PRELOAD_SANDBOX_IMAGE` (default:
  `ghcr.io/nvidia/openshell-community/sandboxes/base:latest`; set to an empty value to skip)
- `HELM_K3S_COLLECTOR_IMAGE` (default:
  `mcr.microsoft.com/dotnet/aspire-dashboard:latest`)
- `HELM_K3S_COLLECTOR_HEALTH_TIMEOUT` (default: `120` seconds)

### 2. Deploy OpenShell

**Iterative dev** (rebuilds on file changes, recommended during active development):
```bash
mise run helm:skaffold:dev
```

**One-shot deploy** (build once and leave running):
```bash
mise run helm:skaffold:run
```

**Supervisor sidecar topology** (build once and leave running):
```bash
mise run helm:skaffold:run:sidecar
```

**Supervisor sidecar topology with TLS/mTLS enabled** (build once and leave running):
```bash
mise run helm:skaffold:run:sidecar-mtls
```

Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
chart. The sidecar profile renders an `openshell-network-init` init container for
nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID, which
must be at least `1000` and distinct from the workload UID. The
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. The default Skaffold values export gateway and Kubernetes-driver traces to
the collector service installed by `helm:k3s:create`. Envoy Gateway opt-in; see the
Optional Add-ons section below.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
the unified local forwarding task:

```bash
mise run helm:k3s:forward
```

The task forwards OTLP/gRPC to `http://127.0.0.1:4317` and the trace UI to
`http://127.0.0.1:18888`. When Skaffold has deployed a Kubernetes gateway, it
also forwards the gateway to `http://127.0.0.1:8090`; otherwise it continues
with the collector ports only. A successful plaintext `helm:skaffold:run` or
`helm:skaffold:run:sidecar` registers the gateway under the worktree-specific
k3d cluster name and selects it as the active gateway. Keep the forwarding
task running while using those endpoints.

### Viewing local traces

The gateway exports OTLP/gRPC to
`http://openshell-collector.observability.svc.cluster.local:4317` through the
default Skaffold values. Forward OTLP/gRPC and the trace UI to the host:

```bash
mise run helm:k3s:forward
```

Open `http://127.0.0.1:18888` and exercise the gateway to inspect gateway and
Kubernetes compute-driver spans under their distinct service names, along with
Agent Sandbox controller reconciliation spans linked through the Sandbox
trace-context annotation. The same command exposes OTLP/gRPC on
`http://127.0.0.1:4317` and, when deployed, the Kubernetes gateway on
`http://127.0.0.1:8090`. The local `gateway:docker`, `gateway:podman`, and
`gateway:vm` tasks detect the collector listener at startup and enable trace
export only while it is reachable.

**HA test deploy** (two gateway replicas + external PostgreSQL Secret): uncomment
`#- ci/values-high-availability.yaml` in `deploy/helm/openshell/skaffold.yaml`,
create the Secret named `openshell-ha-pg` with a `uri` key, then run
`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.

### TLS behaviour

`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
plaintext by default. To test sidecar topology with TLS enabled, use
`mise run helm:skaffold:run:sidecar-mtls`.

| Mode | `server.disableTls` | Gateway scheme |
|------|---------------------|----------------|
| Skaffold dev (default) | `true` | `http://` |
| TLS enabled | `false` (or omitted) | `https://` |

### Connecting through the forwarding task

Port `8080` is already bound by the k3d load balancer when Envoy Gateway is
active, so the forwarding task uses local port `8090` for the gateway. In a
second terminal, confirm that the gateway is registered and active:

```bash
openshell gateway list
```

**Plaintext (default Skaffold deploy):**

```bash
openshell sandbox list
```

**With mTLS enabled** — extract the client cert the PKI hook wrote to the cluster,
then place it where the CLI expects it. Run once after each fresh install:

```bash
mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key
```

The server cert SANs include `localhost` and `127.0.0.1`, so hostname verification
passes over a port-forward without any extra flags:

```bash
openshell sandbox list --gateway-endpoint https://localhost:8090
```

---

## Teardown

### Remove the Helm releases (keep cluster)

```bash
mise run helm:skaffold:delete
```

For a sidecar-profile deployment:

```bash
mise run helm:skaffold:delete:sidecar
```

### Delete the cluster entirely

```bash
mise run helm:k3s:delete
```

This removes the k3d cluster and all resources. Kubeconfig context is left behind
but will point to a deleted cluster — safe to ignore or clean up manually.

---

## Optional Add-ons

Each add-on requires uncommenting the corresponding `valuesFiles` entry in
`deploy/helm/openshell/skaffold.yaml` before running `helm:skaffold:dev` or `helm:skaffold:run`.

### Envoy Gateway (Gateway API / GRPCRoute)

Envoy Gateway is already installed by Skaffold (the `envoy-gateway` Helm release in
`skaffold.yaml`). To activate routing:

1. Uncomment `#- values-gateway.yaml` in `skaffold.yaml`
2. Redeploy: `mise run helm:skaffold:run`
3. Apply the GatewayClass: `mise run helm:gateway:apply`
4. Access: `http://127.0.0.1:8080`

`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and a
`GRPCRoute` in the `openshell` namespace. Envoy Gateway provisions a LoadBalancer
service for the proxy; klipper-lb binds it to hostPort 80, reachable via the
`8080:80` load balancer port mapping.

### Keycloak OIDC

One-time setup — only needed once per cluster lifetime:

```bash
mise run keycloak:k8s:setup
```

This deploys Keycloak (`quay.io/keycloak/keycloak:24.0`) into the `keycloak` namespace,
imports the openshell realm from `scripts/keycloak-realm.json`, and prints a port-forward
command for acquiring tokens from the CLI.

Then activate OIDC in the OpenShell Helm chart:
1. Uncomment `#- ci/values-keycloak.yaml` in `skaffold.yaml`
2. Redeploy: `mise run helm:skaffold:run`

To remove Keycloak:
```bash
mise run keycloak:k8s:teardown
```

### SPIRE / SPIFFE Provider Token Grants

Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate
SPIFFE JWT-SVIDs for dynamic provider token grants:

1. Uncomment the `spire-crds` and `spire` releases in `deploy/helm/openshell/skaffold.yaml`
2. Uncomment `#- ci/values-spire.yaml` in the OpenShell release values files
3. Redeploy: `mise run helm:skaffold:run`

`ci/values-spire-stack.yaml` configures the local SPIRE trust domain as
`openshell.local` and adds a `ClusterSPIFFEID` that maps sandbox pod
annotations to `spiffe://openshell.local/openshell/sandbox/<sandbox-id>`.
OpenShell mounts the SPIFFE CSI Workload API socket at
`/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token
grants. Supervisor-to-gateway authentication remains on the Kubernetes
ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected
Kubernetes compute driver validates the projected token before the gateway
mints its JWT.

---

## Cluster Lifecycle (stop/start)

Stop the cluster without losing state (faster than delete/recreate):
```bash
mise run helm:k3s:stop
mise run helm:k3s:start
```

Check cluster status:
```bash
mise run helm:k3s:status
```

---

## Helm Chart Checks

Run the chart lint task before changing Helm templates, values overlays, or
Skaffold inputs:

```bash
mise run helm:lint
```

If Helm reports missing chart dependencies, remove the specific stale subchart
archive or directory named by the error from `deploy/helm/openshell/charts/`,
then rerun the lint task.

For example, when lint reports `chart metadata is missing these dependencies:
postgresql`, remove stale PostgreSQL chart artifacts:

```bash
rm -f deploy/helm/openshell/charts/postgresql-*.tgz
rm -rf deploy/helm/openshell/charts/postgresql
mise run helm:lint
```

The `charts/` directory is ignored and regenerated by `helm dependency build`
for dependencies still declared in `Chart.yaml`.

---

## Key Files

| Path | Purpose |
|------|---------|
| `deploy/helm/openshell/skaffold.yaml` | Skaffold config — images, Helm releases, values overlays |
| `deploy/helm/openshell/values.yaml` | Default Helm values |
| `deploy/helm/openshell/ci/values-skaffold.yaml` | Dev overrides (image pull policy, TLS disabled for local Skaffold) |
| `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass for Envoy Gateway (`mise run helm:gateway:apply`) |
| `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |
| `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy + realm import |