# Install Zymtrace Backend > Use when installing the zymtrace backend (the AI optimization platform that ingests CPU/GPU profiling data). Covers Kubernetes (Helm) and single-node Docker Compose. Handles license setup, choosing in-cluster vs external ClickHouse/Postgres/object storage, ingress with gRPC and TLS, and air-gapped installs via a custom image registry. Trigger phrases: "install zymtrace", "install zymtrace backend", "deploy zymtrace", "set up zymtrace on kubernetes", "set up zymtrace on EKS / GKE / AKS / on-prem", "helm install zymtrace", "docker compose zymtrace", "stand up the zymtrace platform", "first zymtrace install", "deploy the backend services". - Skill: `zystem-io/install-zymtrace-backend` (Agent Skill, multi-file: 6 files) - Install (CLI): `npx skillmds@latest add zystem-io/install-zymtrace-backend` - Raw SKILL.md: https://api.skillmd.com/api/skills/zystem-io/install-zymtrace-backend/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: zystem-io (https://skillmd.com/u/zystem-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/zystem-io/install-zymtrace-backend --- # Install zymtrace Backend Helps the user install the zymtrace **backend** โ€” gateway, ingest, web, symdb, identity, UI, migrate, plus data stores (ClickHouse, Postgres, S3-compatible object storage). > The profiler agent is a separate install (`install-zymtrace-profiler` skill). This skill is only the backend that receives data. Deep details, secret-creation commands, Docker Compose, and air-gapped live in `${CLAUDE_PLUGIN_ROOT}/skills/install-zymtrace-backend/reference.md` โ€” read it when the decision tree points you there. ## Greet the user (start here) Before any commands or questions, open with a warm welcome. Adapt to context, but always cover: a thank-you, the support channels, and a quick map of what's coming. Sample: > ๐Ÿ‘‹ Thanks for choosing **zymtrace**! I'll walk you through installing the backend โ€” the platform that ingests your CPU/GPU profiling data, surfaces optimization insights, and serves the UI. > > **If you get stuck at any point, reach out:** > - Community Slack: > - Email: > - Sign up / GPU trial license: > > **Tip โ€” analyze GPU and CPU flamegraphs via MCP:** once zymtrace is running, connect the zymtrace MCP to your agent (Claude Code, Codex, or Cursor) โ€” see [`configure-zymtrace-mcp`](../configure-zymtrace-mcp/SKILL.md) โ€” and analyze GPU + CPU flamegraphs in natural language. Docs: > > **Here's the plan:** > 1. Verify your tools (`helm`, `kubectl`) and resolve cluster / namespace. > 2. Check whether you already have a values file โ€” if not, we'll build one together. > 3. Install, verify, then start a quick port-forward so you can see the UI right away. > 4. Decide on long-term exposure (NodePort / ALB / NGINX / Cloud LB). > 5. Hand off to the profiler install so you have data to look at. > > Ready when you are โ€” let's start. Trim the greeting if the user has already given you specifics (cluster, values file, target version). Always include the support links **once** per session. ## Sources of truth (never invent keys) - Live chart `values.yaml` (every key is documented inline): - Docs: (no public source repo โ€” fetch URLs) - Latest release + what changed: (newest version at the top). Use this as the **fallback** to name the latest version when `helm search` can't (repo not added yet, stale cache, air-gapped). - Full URL map: [`shared/references.md`](../../shared/references.md) ## Pre-flight: verify the tools **Do not run any install command until both binaries are confirmed installed.** ##### Claude runs ```bash helm version --short && kubectl version --client kubectl cluster-info | head -2 ``` If `helm` is missing โ†’ point user to . Do **not** offer to install it. Same rule for `kubectl` โ†’ . If `kubectl cluster-info` fails, ask the user to set their kubeconfig. ## Check for a customer-provided values file **Before walking the decision tree, ask:** > Did Zymtrace send you a values file (often named `custom-values.yaml`, `backend-values.yaml`, or `-values.yaml`)? It usually contains your license, DB modes, and other pre-agreed settings. If yes โ†’ read it, skip any decision-tree question whose answer is already set in the file, and go directly to Step 4 with `-f `. Full policy: [`shared/conventions.md` ยง The single values file](../../shared/conventions.md#the-single-values-file). If no โ†’ walk the decision tree below; at the end, write the result to `./custom-values.yaml` and tell them to commit it to source control. ## Pre-resolve what you can Before asking the user any question, resolve from the environment. Only ask when a check fails or returns ambiguous output. > **Namespace + release name.** **Recommend `zymtrace` / `backend`** โ€” these are the defaults in every doc and example, keep them unless the user has a specific reason to deviate. If a release already exists on the cluster, use *its* namespace and name. Full policy: [`shared/conventions.md`](../../shared/conventions.md). Commands below use `` and `` as placeholders; resolve before running. | Variable | Resolve by | |---|---| | Current cluster context | `kubectl config current-context` | | Existing zymtrace release? | `helm list -A \| grep -i zymtrace` | | Latest chart version | `helm search repo zymtrace/backend --versions \| head -3` (after `helm repo add`) | | Ingress controllers present | `kubectl get ingressclass` | | Metrics-server (HPA prereq) | `kubectl top nodes` โ€” returns metrics if installed, errors `Metrics API not available` if not. On EKS, do **not** confuse with `v1.metrics.eks.amazonaws.com` (EKS extension API, doesn't satisfy HPA). | | Default storage class | `kubectl get sc` | Things you **must** ask: - Tier: free CPU-only / GPU trial / paid (licensing). - DB modes for each of ClickHouse / Postgres / object storage. - Domain + TLS arrangement (if production). - Auth type and IdP details if OIDC. - Scale tier (agent count + retention). - Private/air-gapped registry? ## Blockers vs recommendations (don't conflate) Tone matters. Frame findings precisely so the customer doesn't think they have an outage when they don't. **Blockers** (stop the install, surface, ask the user to resolve first): - `helm` or `kubectl` not installed. - `kubectl cluster-info` fails. - Referenced Kubernetes secret missing (chart will fail). - Values file fails `helm template` (schema error / invalid YAML). - Previous release is stuck in an in-progress state. **Recommendations** (note in one short line, proceed): - **Metrics-server not installed** with `hpa.enabled: true` โ†’ install succeeds, HPAs just won't scale. Customer can install metrics-server later. - **License key inline** in values file โ†’ fine for dev/PoC. Recommend `licenseKeySecretName` for prod, don't block. - **`auth.type: none`** โ†’ trusted-network installs run this way on purpose. - **No ingress configured** โ†’ ClusterIP + port-forward is a valid temporary state; the expose skill adds ingress whenever the customer is ready. Rule of thumb: if the operation will succeed but something is suboptimal, that's a **recommendation**, not a blocker. Phrase it as one short line, not a multi-bullet alarm. ## Decision tree Walk these in order. Don't guess defaults โ€” fetch the chart `values.yaml` when uncertain. ### 1. Platform | Answer | Path | |--------|------| | Single node / eval / laptop | โ†’ see [reference.md ยง Docker Compose install](reference.md#docker-compose-install). Stop here, don't continue this tree. | | Kubernetes (prod, staging, on-prem, GKE/EKS/AKS) | โ†’ continue. | If unsure, default to Kubernetes โ€” Docker Compose has no HA/ingress story. ### 2. License - Free CPU-only tier needs **no key**. - GPU trial / Paid: ask where to put it (inline vs Kubernetes secret). Recommend secret for prod. - Where to get a trial key, tier table, and placement details: [reference.md ยง License placement](reference.md#license-placement). ### 3. Databases zymtrace needs ClickHouse, Postgres, and S3-compatible object storage. Each has a `mode`: | Mode | When to use | |------|------------| | `create` (default) | In-cluster, chart-managed. Fastest path. Fine for dev/PoC, viable for small prod. | | `use_existing` | External (ClickHouse Cloud, on-prem CH, RDS, AWS S3, GCS, MinIO). Pick this if the org already runs managed DBs. | | `aws_aurora` / `gcp_cloudsql` | Postgres only. IAM-authenticated. Requires IRSA (Aurora) or Workload Identity (CloudSQL). | Ask which mode per service. Assemble from [`values/k8s-external-dbs.yaml`](values/k8s-external-dbs.yaml). > โš ๏ธ Postgres `secure: true` needed for TLS-enforced managed Postgres. ClickHouse `use_existing.host` MUST be the scheme + HTTP port (`https://host:8443`) โ€” **not** native 9000. Only the HTTP interface is supported. ### 4. Network exposure Two paths: - **Install without exposure** โ†’ leave `ingress.enabled: false` (default). Gateway becomes `ClusterIP`-only; the user can reach it via `kubectl port-forward` for verify, then add exposure later. - **Install with exposure already configured** โ†’ hand off to the [`expose-zymtrace-backend`](../expose-zymtrace-backend/SKILL.md) skill for the exposure decision (NodePort / LoadBalancer / NGINX Ingress / ALB Ingress). That skill will edit the same canonical values file in place before this install proceeds. Either way, exposure can be added or changed at any time via the expose skill โ€” it doesn't have to be locked in at install. ### 5. Auth - `none` โ€” open access; trusted networks only. - `local` โ€” built-in user/password + admin user; requires Ed25519 signing keys. - `oidc` โ€” Google / Okta / Auth0 / Azure AD. Need `clientId`, `clientSecret`, `issuerUri`, registered redirect URI. > **Don't propose `basic`** โ€” deprecated and being removed. Prod default: `oidc` if they have an IdP, else `local` with admin password from a secret. ### 6. Scale How many agents will report, and what retention? Drives ClickHouse sizing more than anything else. | Scale | Template | |-------|----------| | < 20 agents, < 14d | `k8s-minimal.yaml`. Defaults fine. | | 20โ€“100 agents, mixed CPU/GPU, 30d | [`values/k8s-large-scale.yaml`](values/k8s-large-scale.yaml). Bumped CH (500Gi, 2โ€“6 CPU, up to 16Gi mem), tuned probes, HPA scale-down. | | 100+ agents, multi-region, long retention | Combine `k8s-large-scale.yaml` + `k8s-external-dbs.yaml`. Externalize ClickHouse. | Storage rule of thumb: **~5Gi per agent per 30 days** for mixed CPU+GPU. GPU-heavy agents produce ~5โ€“8ร— the events of CPU-only. ### 7. Air-gapped / private registry If mentioned: see [reference.md ยง Air-gapped install](reference.md#air-gapped-install). Mirror `zymtrace-pub-backend`, `zymtrace-pub-ui` (plus DB images if `mode: create`), then set `global.imageRegistry` + `global.appImageRegistry`. --- ## Kubernetes install (Helm) ### Prerequisites - Kubernetes 1.20+, Helm 3.x. - Metrics Server (only if HPA enabled โ€” verify with `kubectl top nodes`, NOT just by listing APIServices on EKS). - CNI with NetworkPolicy enforcement (Calico/Cilium/Weave). Plain Flannel โ†’ set `services.activateNetworkPolicies: false`. ### Step 1: Add the Helm repo ##### Claude runs ```bash helm repo add zymtrace https://helm.zystem.io helm repo update helm search repo zymtrace/backend --versions | head -5 ``` **Version to install.** `helm search` (above) is the source of truth for available chart versions โ€” install the newest it lists unless the user pins one. **If it returns nothing** (repo just added and cache is stale, or air-gapped so the repo isn't reachable), fall back to the changelog at โ€” the top entry is the latest release. Either way, tell the user which version you're about to install (and one line of what's new) before Step 4; they can pin an older one with `--version `. ERROR: `not a valid chart repository` โ†’ network/proxy issue, or air-gapped. Switch to the air-gapped path. ### Step 2: Pick a values template and customize Pick one of these install bundles as the starting point for the customer's canonical values file: - [`values/k8s-minimal.yaml`](values/k8s-minimal.yaml) โ€” chart-managed DBs, NodePort, no auth. Fastest path for dev. - [`values/k8s-external-dbs.yaml`](values/k8s-external-dbs.yaml) โ€” external ClickHouse + Postgres + S3 (incl. Aurora / CloudSQL). - [`values/k8s-large-scale.yaml`](values/k8s-large-scale.yaml) โ€” 100+ agents, 30d retention, bumped CH, tuned probes. Copy to the canonical filename (e.g. `zymtrace-custom-values.yaml` if the customer doesn't have one) and edit placeholders. **Need ingress now?** Run the [`expose-zymtrace-backend`](../expose-zymtrace-backend/SKILL.md) skill โ€” it'll edit the same canonical file in place โ€” then come back to Step 3 here. ### Step 3: Create secrets ##### What you need to do in a terminal Secrets must be created by the user โ€” never write license keys, OIDC client secrets, admin passwords, or DB passwords into the conversation or values files. Create only the secrets your values file references. Command reference: [reference.md ยง Creating secrets](reference.md#creating-secrets). Typical minimum for prod: `zymtrace-license`, plus one of `oidc-creds` / `zymtrace-admin`, plus `zymtrace-signing-keys` (auth=local/oidc). The chart does **not** pre-check that referenced secrets exist โ€” missing secrets surface as `CrashLoopBackOff` in Step 5. ### Step 4: Install > `` and `` are placeholders for the resolved release name + namespace. **Recommended defaults: `backend` / `zymtrace`** โ€” use them unless the user has a specific reason to deviate. Substitute before running. ##### Claude runs ```bash helm upgrade --install zymtrace/backend \ --namespace --create-namespace \ -f .yaml \ --reset-then-reuse-values \ --atomic --debug ``` `--reset-then-reuse-values` is **mandatory on every run for this chart**, including first install. Why: [reference.md ยง Why always reset-then-reuse-values](reference.md#why-always-reset-then-reuse-values). `--atomic` rolls back on failure (avoids half-deployed state). `--debug` prints rendered manifests. ERROR: `failed pre-install: timed out` โ†’ usually a missing secret. Run `kubectl get events -n --sort-by=.lastTimestamp | tail -20`, fix the secret per Step 3, re-run. ERROR: `another operation โ€ฆ in progress` โ†’ a previous helm op is stuck. `helm history -n `; resolve with `helm rollback` or `helm uninstall` (destructive โ€” confirm with the user first). ### Step 5: Verify ##### Claude runs ```bash bash ${CLAUDE_PLUGIN_ROOT}/skills/install-zymtrace-backend/scripts/verify-backend.sh ``` If the user has overridden `global.namePrefix`, also pass it: `PREFIX= bash ${CLAUDE_PLUGIN_ROOT}/skills/install-zymtrace-backend/scripts/verify-backend.sh `. Runs `helm status`, dumps `kubectl get` for pods/jobs/svc/ingress/hpa, dumps logs for each backend service, describes any non-Running pod. Use the Done checklist below as exit criteria โ€” do not declare success until every box checks. ### Step 6: Persist the canonical values file ##### Claude runs ```bash # Back up first if the file already exists (won't on a true greenfield install) [ -f ] && cp .bak.$(date +%Y%m%d-%H%M%S) helm get values -n > ``` `` is whatever name the customer gave you in ยง0 / Pre-resolve (e.g. `acme-zymtrace.yaml`). If they didn't provide one, default to `zymtrace-custom-values.yaml` โ€” that's now their canonical file. See [`shared/conventions.md`](../../shared/conventions.md) for the rules on respecting customer filenames and backing up before writing. If a backup was created, tell the user the path. Recommend they commit the new canonical file: `git add && git commit -m "zymtrace: capture / values after install"`. ### Step 7: Quick-win port-forward (let them see the UI now) Once Done is satisfied, **offer to port-forward immediately** โ€” this is the fastest way to give the customer a visible result before they commit to a long-term exposure decision. > Want me to start a port-forward right now so you can open the UI? It runs in the background; you can switch to a permanent exposure (NodePort / LoadBalancer / Ingress) afterwards. If yes: ##### Claude runs ```bash kubectl port-forward -n svc/-gateway 8080:80 > /tmp/zymtrace-pf.log 2>&1 & echo "port-forward PID: $!" sleep 2 curl -fsI http://localhost:8080 | head -1 # sanity check ``` Then tell them: > The UI is at . To stop the port-forward later: `pkill -f 'kubectl port-forward.*-gateway'`. If port 8080 is taken, change it to `8081:80` (or any free local port). If no, skip to Step 8. ERROR: `Unable to listen on port 8080: bind: address already in use` โ†’ pick another local port. Re-run with `8081:80` etc. ### Step 8: Ask how they want to expose the backend long-term **Always ask** โ€” don't assume port-forward is good enough. | Option | When it fits | |--------|-------------| | **1. NodePort** | PoC, on-prem, no cloud LB available | | **2. AWS ALB Ingress** (HTTPS via ACM) | EKS โ€” internal or internet-facing | | **3. NGINX Ingress** (TLS via cert-manager) | On-prem / non-AWS, or already running NGINX | | **4. Cloud LoadBalancer** (direct NLB/CLB) | Rare; ask what they want before suggesting | If they prefer to stick with port-forward for now โ†’ skip to Step 9. For any of 1โ€“4 โ†’ hand off to the [`expose-zymtrace-backend`](../expose-zymtrace-backend/SKILL.md) skill, which edits the canonical values file in place and applies via `helm upgrade --install`. Ask them to provide the hostname themselves โ€” don't suggest a specific pattern. Once that completes, return here for Step 9. ### Step 9: Hand off to profiler install The backend has no data until a profiler agent reports to it. Suggest the `install-zymtrace-profiler` skill next. --- ## Done Exit when ALL of the following are true (substitute `` / `` / ``): - [ ] `helm status -n ` reports `STATUS: deployed`. - [ ] All pods in the `` namespace are `Running`. - [ ] Migration succeeded โ€” either `-migrate` Job is at `1/1` succeeded, **or** the Job is absent (Helm `pre-install` hook auto-deletes on success). With `helm status STATUS: deployed`, absence is the expected state, not a failure. - [ ] `-gateway` service exists; if `ingress.enabled=true`, an Ingress object also exists. - [ ] No license / auth / forbidden errors in `kubectl logs deployment/-ingest -n --tail=50`. - [ ] Gateway responds: `curl -fsI http://` returns anything except connection-refused / timeout / 5xx. If any box fails, hand off to the `troubleshoot-zymtrace-backend` skill, or use the `verify-backend.sh` output. ## Common pitfalls - **NGINX/Traefik missing `backend-protocol: "GRPC"`** โ†’ agents can't push profiles; UI may still work. - **ALB set to GRPC backend** โ†’ HTTP/1.1 clients get HTTP 464. Use HTTP + HTTP2. See [`expose-zymtrace-backend` ยง ALB HTTP2 quirk](../expose-zymtrace-backend/reference.md#alb-http2-quirk). - **`proxy-body-size` unset / too small** โ†’ symbol uploads fail silently. - **NodePort + OIDC** โ†’ `redirectUri` must be set explicitly (chart can't auto-derive) and registered with the IdP. - **ClickHouse `use_existing.host` on native port `9000`** โ†’ ingest crashes. Use HTTP `8123`/`8443`. - **`auth.admin.password=admin`** in prod โ†’ use `passwordSecretName`. - **HPA on without metrics-server** (recommendation, **not a blocker**) โ†’ install succeeds; HPAs sit at `/80%` and stay at `minReplicas`. Truth check: `kubectl top nodes`. To enable scaling later: `kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml`. On EKS, `v1.metrics.eks.amazonaws.com` is **not** the metrics-server HPA needs โ€” don't be fooled by it. --- ## Security constraints - **Never** write a raw license key, OIDC client secret, admin password, or DB password into any values file, session message, or commit. Use `*SecretName` / `*SecretKey` and create the secret imperatively with `kubectl create secret`. - **Never** generate a Kubernetes `Secret` YAML manifest from this skill โ€” always `kubectl create secret generic โ€ฆ` so values never land on disk. - **Never** use namespace `default` for zymtrace resources. Always pass `--namespace ` with an explicit (non-`default`) namespace. - **Never** run `helm uninstall`, `kubectl delete namespace`, `kubectl delete pvc`, or any operation that drops persistent data without explicit user confirmation. - **Never** propose `auth.type: basic` โ€” deprecated. - **Never** issue `helm upgrade` or `helm upgrade --install` for this chart without `--reset-then-reuse-values` โ€” even on first install, even with `-f`. The flag is harmless on a fresh install and is the only guard against silent data loss on subsequent partial `--set` upgrades. - **Never** disable TLS in production (`--disable-tls` is fine for dev/NodePort only). - **Never** skip Step 5 verification โ€” pods can be `Running` while ingest is silently rejecting profiles.