Gitops Tenant Onboarding

GitOps Tenant Onboarding

Onboarding a tenant — an application that runs on a shared, multi-tenant Flux cluster from its own repository — follows the same recurring shape every time. The tenant repo builds a container image and publishes its Kubernetes manifests as a signed OCI artifact; the platform pulls that artifact with a Flux OCIRepository + Kustomization and runs it in a dedicated, locked-down namespace. This skill is the agent-facing companion to that pattern: the decisions, the conventions, and — most usefully — the gotchas that make a fresh tenant fail if you miss them.

It is built around an opinionated but industry-standard stack — Flux, External Secrets Operator backed by OpenBao/Vault, Kyverno + PodSecurity, Cilium / Gateway API, cosign — so the steps transfer to any cluster wired the same way; adapt the resource names to your platform.

The two halves

Onboarding always spans two repos, and a tenant is not live until both land:

1. The tenant repo — created from a tenant template. Ships the shared, framework-agnostic CI/CD plumbing (build → signed publish → release) plus a deploy/ directory of Kubernetes manifests you own and customise.
2. The platform registration — a small directory in the platform repo (apps/<tenant>/) that grants the tenant a namespace, an identity (a ServiceAccount), RBAC, a network policy, the OpenBao-backed image-pull secret, and the Flux resources that pull and verify the tenant's artifact.

Open each as a PR. The tenant goes live when the platform registration merges and Flux reconciles it — the tenant repo alone does nothing until it is registered.

Tenant repo: the deploy/ manifests

A tenant's deploy/ is a Kustomize overlay. The full set, and what each is for:

Conventions and gotchas that bite

These are the failure modes worth memorising — each one produces a tenant that looks configured but is broken at reconcile, admission, or runtime:

• The container name MUST equal the repository name. The signed-publish pipeline pins the freshly built image digest into the container named after the repo. A mismatch means the running image is never updated. Rename the placeholder container throughout deployment.yaml.
• Set the seccompProfile at the pod level, not only the container. A PodSecurity "restricted" cluster (and a Kyverno require-seccomp-profile rule) demands securityContext.seccompProfile.type: RuntimeDefault on the pod spec. Setting it only on the container passes naive review but is rejected at admission. Set runAsNonRoot: true + seccompProfile on both pod and container; the container also gets allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities.drop: [ALL].
• A default-deny cluster makes a tenant unreachable until it ships an allow policy. If the platform generates a deny-all network policy in every tenant namespace, a tenant with no networkpolicy.yaml cannot receive Gateway traffic or reach its own database. Ship the allow policy day-one: ingress from the gateway on the app port, egress to DNS, plus intra-namespace + operator rules if it has a database. (With Cilium, an empty endpoint selector {} in a fromEndpoints/toEndpoints rule selects this namespace's own pods — so intra-namespace rules need no namespace name and stay placeholder-free.)
• Tenant secrets come from the secret store, never SOPS. No tenant ships an encrypted Secret in git. App secrets are delivered by External Secrets from OpenBao/Vault (see below). The Flux Kustomization needs no spec.decryption.
• Use a namespaced SecretStore, never the shared ClusterSecretStore. A multi-tenant cluster blocks tenants from referencing the cluster-scoped store (a Kyverno restrict-tenant-secret-stores-style policy) so one tenant can't read another's path. The namespaced store authenticates via the tenant's own Vault role, scoped to apps/<tenant>/*. The single carve-out is the platform-managed image-pull secret (below), applied by the GitOps controller, not the tenant.
• maxUnavailable: 1, not minAvailable: 1, in the PDB. maxUnavailable: 1 is drain-safe at every replica count — at one replica the pod can still be evicted (no deadlock), at 2+ it gives rolling protection. A minAvailable: 1 PDB over a single replica permits zero voluntary evictions and wedges every node drain (autoscaler recycles, rolling reboots).
• If you start under the HA replica floor, opt out explicitly. A cluster that audits for a minimum replica count (e.g. 3) will flag a fresh single-replica tenant. Carry the platform's exemption label day-one to stay clean in policy reports; delete it and raise replicas when you want HA.
• The HTTPRoute attaches to the shared gateway via parentRefs (name/namespace/sectionName), not a per-tenant gateway. Set the hostnames to the tenant's real host and the backendRefs to the tenant Service.

Secrets: app secrets vs. the image-pull secret

Two different mechanisms, often confused:

• App secrets (DB creds, API keys) — tenant-owned end-to-end. The platform provisions only the namespaced SecretStore + the Vault role/policy (scoped to apps/<tenant>/*, read and write so the tenant can seed); it never seeds a tenant's app values. How a value reaches the path is the tenant's business: paste an externally-issued credential straight into OpenBao, or seed a generated value in-cluster with a Password generator → PushSecret (refreshInterval: "0") → ExternalSecret. The only hard rules: nothing sensitive sits in git in plaintext, and workloads read values from the store via ExternalSecrets.
• The image-pull secret (ghcr-auth / equivalent) — platform-managed, not a tenant secret. The registration dir ships an ExternalSecret that sources the shared registry pull credential from the cluster-scoped store and materialises the dockerconfigjson the OCIRepository and ServiceAccount consume. It may use the ClusterSecretStore precisely because the GitOps controller (not the tenant SA) applies it — the policy carves out controller-applied resources.

Platform registration: apps/<tenant>/

Copy an existing tenant directory and rename. The resource set:

Then add <tenant>/ to the apps kustomization.yaml. For a tenant that runs its own external-dns for a custom domain, add the extra grants (an external-dns-rbac.yaml binding the tenant external-dns SA to the tenant-scoped ClusterRoles, and an FQDN-pinned external-dns-networkpolicy.yaml) — mirror a tenant that already does this rather than inventing the RBAC.

In sync.yaml, set the artifact url (oci://<registry>/<tenant>/manifests) and keep the verify block pointed at the trusted publish-workflow identity, so only artifacts produced by that workflow are ever reconciled.

Publishing & trust

On every release tag, the tenant's CD calls the platform's signed-publish workflow: it builds and pushes the image, pins the digest into deployment.yaml, pushes the manifests as an OCI artifact, and cosign-signs both (keyless, via CI OIDC). The platform's OCIRepository verifies that signature against the publish-workflow identity — the trust root that ensures only artifacts from the trusted pipeline reach the cluster. Tags come from Conventional-Commit merges to main driving semantic-release, so a normal merge produces a publish automatically.

Validate before you open the PR

• Render + schema-validate the tenant deploy/ and the platform registration: kubectl kustomize <dir> | kubeconform -strict (with built-in schemas + a pinned CRD catalog for the operators in use). This catches a broken or schema-invalid manifest before CI.
• Walk the gotcha list above against the diff — container-name == repo-name, pod-level seccomp, an allow network policy present, namespaced (not cluster) SecretStore, maxUnavailable PDB, replica-floor handled, HTTPRoute parent + hostname set.
• Confirm both halves exist — a tenant repo with no platform registration (or vice versa) is a half-onboarded tenant that will never reconcile.

Staying current

The tenant template keeps the shared plumbing current via template-sync: it opens a PR in the tenant whenever a pinned action, a workflow, or a convention changes. Review and merge it like any dependency update — the files the tenant owns (deploy/, app code, CI) are listed in .templatesyncignore and never touched.