Loom Kustomize

Kustomize Skill

Overview

Template-free, declarative Kubernetes config management via merging. kubectl apply -k <dir> and kustomize build <dir> render an overlay. Failure modes here are overwhelmingly silent (wrong output, exit 0) — always diff the kustomize build output before/after any structural change.

Core Concepts

Directory Layout

k8s/
├── base/                    # shared, env-agnostic (kustomization.yaml + resources)
├── overlays/
│   ├── dev/                 # low resources, debug
│   ├── staging/             # + monitoring component
│   └── prod/                # HA, security, digest-pinned images
└── components/              # opt-in features: monitoring, ingress, debug-tools

Keep the hierarchy shallow (base → overlay, not base → overlay1 → overlay2). Paths in resources: are relative to the kustomization.yaml's own directory.

Quick Reference

Generators (configMapGenerator / secretGenerator items): literals: (K=v), files: (app.properties, or alias=path), envs: (config.env). Secrets add type: (e.g. kubernetes.io/tls). behavior: = create (default) | merge | replace.

Transformers (top-level keys): namespace, namePrefix, nameSuffix, labels (use over deprecated commonLabels), commonAnnotations, images (newName/newTag/digest), replicas (- name: x count: n), replacements.

Patch Strategies

Both go under the unified patches: field, which auto-detects type from content and whose target supports group/version/kind/name/namespace + labelSelector/annotationSelector (one patch → many resources). patchesStrategicMerge/patchesJson6902 are deprecated (v5.0.0) and gone from the v1 API.

Components

Reusable opt-in bundles for features only some environments need (monitoring, ingress, debug tools, security hardening). Use a patch instead for required, always-applied, or env-specific value tweaks.

# k8s/components/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1   # alpha — API may change without deprecation window
kind: Component
resources:
  - servicemonitor.yaml
patches:
  - path: patch-metrics.yaml
    target: {kind: Deployment}
labels:
  - pairs: {prometheus.io/scrape: "true"}
    includeSelectors: false

Reference Components under components:, never resources:. A Component placed under resources: is silently misapplied as a plain manifest; kustomize enforces the split (Components rejected from resources:, Kustomizations from components:). Components run against the parent's accumulated resource set and can be nested.

Examples

Base kustomization + Deployment

# k8s/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp

# Use the labels transformer, NOT deprecated commonLabels (which mutates the
# IMMUTABLE spec.selector.matchLabels → delete/recreate on any later change).
# includeSelectors defaults false (safe on live resources); includeTemplates
# reaches pod templates. Define real selector labels directly in deployment.yaml.
labels:
  - pairs: {app: myapp, managed-by: kustomize}
    includeSelectors: false
    includeTemplates: true

commonAnnotations: {contact: team@example.com}

resources:
  - deployment.yaml
  - service.yaml
  - serviceaccount.yaml

configMapGenerator:
  - name: app-config
    literals: [LOG_LEVEL=info, MAX_CONNECTIONS=100]

images:
  - name: myapp
    newName: registry.example.com/myapp
    newTag: latest

# k8s/base/deployment.yaml (selector label defined explicitly — not injected)
apiVersion: apps/v1
kind: Deployment
metadata: {name: myapp}
spec:
  replicas: 1
  selector:
    matchLabels: {app: myapp}
  template:
    metadata:
      labels: {app: myapp}
    spec:
      serviceAccountName: myapp
      containers:
        - name: app
          image: myapp
          ports: [{containerPort: 8080, name: http}]
          envFrom: [{configMapRef: {name: app-config}}]
          resources:
            requests: {memory: 128Mi, cpu: 100m}
            limits: {memory: 256Mi, cpu: 200m}
          readinessProbe:
            httpGet: {path: /ready, port: http}
            initialDelaySeconds: 5

Production overlay (annotated)

# k8s/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp-prod

labels:
  - pairs: {environment: prod, criticality: high}
    includeSelectors: false
    includeTemplates: true

commonAnnotations:
  oncall: sre-team@example.com

# SINGLE resources: block. A second resources: key later in the same document
# silently wins (duplicate YAML map keys), dropping ../../base entirely.
resources:
  - ../../base
  - poddisruptionbudget.yaml
  - horizontalpodautoscaler.yaml
  - networkpolicy.yaml

patches:
  - path: patch-resources.yaml     # bump requests/limits
  - path: patch-affinity.yaml      # required podAntiAffinity + nodeAffinity
  - path: patch-security.yaml      # runAsNonRoot, drop ALL, RO rootfs, seccomp

configMapGenerator:
  - name: app-config
    behavior: merge
    envs: [config-values.env]

secretGenerator:
  - name: app-secrets
    envs: [secrets.env]            # gitignored / external-managed

images:
  - name: myapp
    newTag: v1.2.3
    digest: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  # pin by digest in prod

replicas:
  - {name: myapp, count: 5}

components:
  - ../../components/monitoring
  - ../../components/ingress

Overlay patch files are minimal partial docs matched by kind/name, e.g. a security patch:

# patch-security.yaml
apiVersion: apps/v1
kind: Deployment
metadata: {name: myapp}
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile: {type: RuntimeDefault}
      containers:
        - name: app
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: {drop: [ALL]}

JSON 6902 patch (precise array ops)

# kustomization.yaml
patches:
  - path: json-patch.yaml
    target: {group: apps, version: v1, kind: Deployment, name: myapp}

# json-patch.yaml — list of {op, path, value}; auto-detected as JSON 6902
- op: add                                                 # append sidecar
  path: /spec/template/spec/containers/-
  value: {name: log-shipper, image: fluent/fluent-bit:2.0}
- op: replace                                             # container 0 image
  path: /spec/template/spec/containers/0/image
  value: registry.example.com/myapp:v1.2.3
- op: remove                                              # env var by index
  path: /spec/template/spec/containers/0/env/3
- op: test                                                # conditional guard
  path: /spec/replicas
  value: 1
- op: replace
  path: /spec/replicas
  value: 5

Generators

configMapGenerator:
  - name: app-config
    behavior: merge                 # create (default) | replace | merge
    literals: [LOG_LEVEL=debug]     # overrides base key
    files: [app.properties, tls.crt=certs/server.crt]
    envs: [config.env]
    options:
      disableNameSuffixHash: true   # ⚠ defeats config-driven rollouts (see Gotchas)

secretGenerator:
  - name: tls-secrets
    files: [tls.crt, tls.key]
    type: kubernetes.io/tls
  - name: app-secrets
    envs: [secrets.env]             # gitignored — never commit secrets

Replacements (propagate a field value)

replacements:
  # Propagate hashed ConfigMap name into the Deployment volume ref
  - source: {kind: ConfigMap, name: app-config, fieldPath: metadata.name}
    targets:
      - select: {kind: Deployment}
        fieldPaths:
          - spec.template.spec.volumes.[name=config].configMap.name

Remote bases

resources:
  - https://github.com/org/repo//k8s/base?ref=v1.0.0   # pin to tag or full SHA (see Security)

Common Tasks

# Validate / preview
kustomize build k8s/overlays/prod | kubectl apply --dry-run=server -f -
kubectl diff -k k8s/overlays/prod

# Inspect name/hash transformations
kustomize build k8s/overlays/prod | rg '^  name:'
kustomize build k8s/overlays/prod | rg -A2 'kind: ConfigMap'

# Migrate deprecated fields in place (idempotent; wrap in a build-diff)
kustomize edit fix

# Bootstrap kustomization.yaml from existing manifests
kustomize create --autodetect        # or --resources deployment.yaml,service.yaml

GitOps & CI/CD Integration

ArgoCD: Application.spec.source.path: k8s/overlays/prod, syncPolicy.automated {prune, selfHeal}. Flux: kustomize.toolkit.fluxcd.io/v1 Kustomization with spec.path, prune: true, sourceRef (GitRepository), healthChecks.

Both bundle their own kustomize version — pin/match it (see Currency gotcha). Prefer the controller's authenticated Git source over remote bases in resources:.

# CI: render, then validate. kubeval is unmaintained — use kubeconform.
kustomize build "k8s/overlays/${OVERLAY}" > all.yaml
kubeconform -strict -summary -kubernetes-version 1.29.0 all.yaml
conftest test all.yaml    # OPA policy gate

Helm inflation gotcha

helmCharts requires kustomize build --enable-helm. Without the flag the whole helmCharts section is silently skipped (no error), so CI can emit manifests missing entire components yet still "pass". kubectl apply -k CANNOT pass --enable-helm (unknown flag), making helmCharts effectively dead there — use kustomize build --enable-helm | kubectl apply -f -. It's a deliberately limited subset (no private-registry auth, no post-renderers). For anything nontrivial, helm template then kustomize-patch on top.

Troubleshooting

Expert Practices: Idioms, Anti-Patterns & Gotchas

High-signal guidance for production Kustomize. Most of these failure modes are silent — wrong output with exit code 0 — so always diff kustomize build output before and after any structural change.

Anti-Patterns (deprecated fields removed from the v1 API)

commonLabels → labels transformer. commonLabels (deprecated v5.0.0) injects every label into spec.selector.matchLabels in addition to metadata.labels and pod templates. Because a Deployment/StatefulSet/Service selector is immutable after first apply, adding or changing any commonLabel later forces a delete-and-recreate. The labels transformer defaults includeSelectors: false (safe on live resources); add includeTemplates: true so pod labels still propagate (Prometheus scraping, log aggregation). Reserve includeSelectors: true for stable identity labels set only at creation time — and prefer defining those directly in base manifests so the selector contract is explicit. The docs warn that changing selectors on live resources "could result in failures."

labels:
  - pairs:
      app.kubernetes.io/part-of: myplatform
      environment: prod
    includeSelectors: false   # safe on live resources — does NOT touch matchLabels
    includeTemplates: true    # still reaches pod template labels

patchesStrategicMerge / patchesJson6902 → unified patches. Both deprecated in v5.0.0. The patches field auto-detects the type from content (resource-shaped doc = strategic merge; list of {op, path, value} = JSON 6902), and its target supports group/version/kind/name/namespace plus labelSelector and annotationSelector, so one patch can hit many resources. Migration is not behavior-neutral: patches runs after the label transformers while patchesJson6902 ran before them, so a patch that touched /metadata/labels can produce different output post-migration. Always diff.

vars → replacements. vars ($(VAR) substitution, deprecated v5.0.0) is replaced by replacements, a structured source/target model that propagates any field value (including hashed ConfigMap/Secret names). Migration is NOT 1:1: replacements works on field pointers and cannot do free-form mid-string concatenation like path: $(ROOT)/cluster/$(REVISION). It can do single-segment replacement within a string via the delimiter + index options, but for true composite strings, restructure (compose in an init container/env var) before migrating.

replacements:
  - source: {kind: ConfigMap, name: app-config, fieldPath: metadata.name}
    targets:
      - select: {kind: Deployment}
        fieldPaths:
          - spec.template.spec.volumes.[name=config].configMap.name

Idioms

kustomize edit fix is the canonical, idempotent migration tool. It migrates deprecated fields in place — bases→resources, commonLabels→labels, patchesStrategicMerge/patchesJson6902→patches, and (experimentally, --vars) vars→replacements. Run it per kustomization directory; use it as a CI lint to surface remaining deprecated fields. Two things it can't guarantee semantically: the patch apply-order shift and vars mid-string interpolation — so wrap it in a build diff:

kustomize build . > before.yaml
kustomize edit fix
kustomize build . > after.yaml
diff before.yaml after.yaml   # confirm semantic equivalence

Components apply against the accumulated resource set. Components use apiVersion: kustomize.config.k8s.io/v1alpha1, kind: Component (alpha — the API may change without a deprecation window). Unlike a base, a Component runs its patches/generators against the resource set the parent has accumulated so far, enabling additive opt-in feature bundles. They must be referenced under components:, never resources:, and can be nested (a Component may itself reference other Components via components:).

Design with Kustomize's deliberate constraints. The maintainers commit to NOT supporting: (1) removal directives (you cannot delete a label/patch/resource from a base — restructure/fork the base, or use a JSON 6902 op: remove for a specific path); (2) ${VAR} templating; (3) build-time side effects from CLI args or shell env vars (output must be fully determined by version-controlled files); (4) globs. The sanctioned pattern for env-driven output is to mutate the kustomization, commit, then build:

kustomize edit set image myapp=registry.example.com/myapp:${GIT_SHA}
kustomize build overlays/prod

Namespace transformer only re-homes default-named RBAC subjects. Setting top-level namespace: runs the namespace transformer in DefaultSubjectsOnly mode: it only namespaces RoleBinding/ClusterRoleBinding subjects whose name is literally default. A ServiceAccount subject with any other name gets no namespace — silently breaking RBAC — unless you configure a custom NamespaceTransformer with setRoleBindingSubjects: allServiceAccounts. Verify RBAC subjects explicitly after setting a namespace.

Gotchas (silent failures)

Strategic merge patches silently REPLACE entire list fields on CRDs. For built-in types, strategic merge knows array merge keys from the embedded OpenAPI schema (e.g. containers merges on name). Custom Resources have no such schema, so kustomize falls back to JSON merge semantics: a patched list field replaces the whole array with no error — dropping every element not in the patch. Fix with either an openapi field carrying x-kubernetes-patch-merge-key/x-kubernetes-patch-strategy, or a JSON 6902 patch (which behaves identically for built-in and custom resources):

patches:
  - target: {group: argoproj.io, version: v1alpha1, kind: Application, name: my-app}
    patch: |-
      - op: add
        path: /spec/sources/-
        value:
          repoURL: https://github.com/org/repo

disableNameSuffixHash: true defeats config-driven rolling updates — and a global true cannot be overridden. The content-hash suffix (app-config-k8bgk8h4t6) is what triggers a rollout on config change: the new name changes the pod spec, so pods are replaced. Disabling it gives a stable name but silently breaks this — config changes apply to the ConfigMap, but Deployments do not roll. Worse, the boolean follows "global true wins": setting it in top-level generatorOptions strips the hash from every generator, and a per-generator disableNameSuffixHash: false is ignored. Only ever disable per-generator, and only for resources referenced by a fixed external name; if you need a stable name and rolling updates, keep the hash and propagate the hashed name via replacements.

replacements list filters like [name=nginx] are unanchored regex. Since v4.5.3, spec.template.spec.containers.[name=nginx].image matches any element whose name contains nginx — my-nginx, nginx-sidecar — and all matches are targeted, silently overwriting several containers. Anchor explicitly with [name=^nginx$], or use index-based paths. (This differs from a patches target's name/namespace, which ARE auto-anchored.)

configMapGenerator behavior: merge matches by logical name AND namespace. An overlay generator with behavior: merge matches the base by pre-hash name — but if one side specifies namespace and the other omits it, kustomize treats them as different generators and creates a second ConfigMap instead of merging (no error). merge also only merges discrete keys; if both reference the same filename, the overlay file replaces the base file entirely. Keep namespace consistent (or absent) on both sides, and verify the build emits one ConfigMap, not two.

sortOptions only applies to the top-level (build-target) kustomization. sortOptions (order: legacy | fifo; v5.0.0+, replaces --reorder) controls output ordering — legacy (default) places Namespaces/ServiceAccounts/RBAC first, which is why kubectl apply -k usually works without explicit ordering. Instances in bases/components reached via resources are ignored, so set it in the overlay that is the actual build target (what ArgoCD/Flux points to), never in a shared base.

Pin a standalone kustomize version. kubectl embeds an older kustomize that does not track standalone releases (the lag has spanned major versions). Features added after the embedded version — unified patches, sortOptions, Components, newer replacements behaviors — may error or be silently ignored under kubectl apply -k while a dev's standalone v5 produces correct output. ArgoCD/Flux also bundle their own versions. Standardize on a pinned kustomize build | kubectl apply -f - in CI and match the GitOps controller's version. Check with kustomize version and kubectl version --client -o json (kustomizeVersion).

Security

Never use --load-restrictor=LoadRestrictionsNone with untrusted input. The default LoadRestrictionsRootOnly (v2.0) prevents file references (resources, configMapGenerator files) from reading outside the kustomization root — a real security boundary. With it off, a kustomization you don't fully control (e.g. a compromised remote base) can use a configMapGenerator file reference to read arbitrary host files (credentials, TLS keys, /etc/passwd). It also breaks relocatability. Only use it in fully trusted, audited build environments; the legitimate alternative for cross-directory file needs is a proper base with its own kustomization.yaml.

Pin remote bases to a full commit SHA or tag. Kustomize fetches remote bases over the network at build time. ?ref=main (or no ref, which uses default-branch HEAD) means a force-push or compromised upstream flows straight into your cluster on the next reconcile with no change to your repo. Short SHAs are unsupported — use a full fetchable SHA (strongest; tags can be force-pushed) or a versioned tag. Remote fetches also add latency and a rate-limit/availability dependency, so mirror critical bases locally. In GitOps, prefer the controller's own authenticated GitRepository/OCIRepository source.

resources:
  - https://github.com/org/platform-base//k8s/base?ref=a3f8c2d1e4b5f6a7b8c9d0e1f2a3b4c5d6e7f8a9   # full SHA
  - https://github.com/org/platform-base//k8s/base?ref=v2.3.1                                       # or versioned tag

Never commit secrets. Use sealed-secrets / external-secrets-operator / SOPS, or secretGenerator with gitignored env files. Pin production images by digest, not mutable tags. Enforce policy at admission (OPA Gatekeeper, Kyverno).

Currency

Validate with kubeconform, not kubeval. kubeval is unmaintained and its schema registry is stale, so it cannot validate recent Kubernetes API versions and silently passes manifests using newer/renamed fields. kubeconform is the maintained successor on the same JSON-schema approach, with a current registry and CRD support via -schema-location:

kustomize build k8s/overlays/prod | kubeconform -strict -summary -kubernetes-version 1.29.0

Verification Checklist

• kustomize build <overlay> succeeds and output diffed against the prior render for unintended changes.
• No deprecated fields remain (commonLabels/patchesStrategicMerge/patchesJson6902/vars/bases) — kustomize edit fix clean.
• Labels use the labels transformer with includeSelectors: false; selectors defined explicitly in base manifests.
• Exactly one resources: key per file; Components under components: (never resources:).
• ConfigMap/Secret hash suffix intact where rolling updates are wanted; merge generators emit ONE object (consistent namespace).
• CRD list patches use JSON 6902 (or openapi merge keys), not bare strategic merge.
• Prod images pinned by digest; secrets gitignored/external; remote bases pinned to full SHA/tag.
• CI validates with kubeconform (+ policy gate); GitOps controller kustomize version matches CI.