Kubernetes and container standards (staff/principal level). Use when writing or reviewing Dockerfiles/Containerfiles, OCI images, Kubernetes manifests (*.yaml with apiVersion/kind), Helm charts (Chart.yaml, values.yaml, templates/), Kustomize overlays (kustomization.yaml), GitOps configs (Argo CD Application, Flux Kustomization/HelmRelease), admission policies (Kyverno, OPA/Gatekeeper), image signing (cosign) or SBOM tooling. Also for Windows nodes and Windows containers (Server Core, Nano Server, ltsc2022/ltsc2025 base images, host/image version compatibility, runAsUserName, HostProcess, gMSA and GMSACredentialSpec) and for batch scheduling on Kubernetes (job queues, gang scheduling, Kueue, Volcano, ResourceQuota).
Applies when creating or reviewing: Dockerfile/Containerfile, K8s manifests (*.yaml with apiVersion/kind), Helm charts (Chart.yaml, templates/), kustomization.yaml, Argo CD/Flux resources, Kyverno/OPA policies, pipelines that build/sign/deploy images. Does not apply to local development Compose (use proportionate judgement).
Not applicable: see iac-standards (provisioning of the cluster and the infrastructure with Terraform/OpenTofu and Ansible; the boundary is decided there: cluster and platform with TF/Tofu, workloads via GitOps), aws-standards/azure-standards/gcp-standards (the managed control plane —EKS/AKS/GKE—, its integration with IAM/VPC and its cost; here what runs inside), container-runtime-security-standards (what happens after the Pod starts: seccomp, runtime choice and pinning, container escape, runtime detection with Falco/Tetragon/eBPF, drift and node forensics; here admission, Kyverno policies, Pod Security Standards, declarative securityContext and signature verified at admission), selinux-standards (the container's MAC: container_t, MCS, :z/:Z, udica, AppArmor profiles), networking-standards (underlying physical network, VLAN, BGP and MTU; here Services, Gateway API and NetworkPolicy), cicd-standards (the pipeline that builds and signs the image and triggers the deployment; here the resulting manifest and the verification at admission), observability-standards (Prometheus, OTel Collector and their rules; here only probes, resources and the ServiceMonitor), sre-practice-standards (SLOs, capacity and on-call), appsec-standards (application code), vulnerability-management-standards (triage and SLA of the CVEs Grype/Syft report), cryptography-pki-standards (signing key management and internal PKI; here only the use of cosign and cert-manager), identity-access-management-standards (IdP and federation; here cluster RBAC and ServiceAccounts), homelab-standards and onprem-standards (single-node k3s/Talos and the hardware/OS underneath), air-gapped-standards (the internal registry/mirror and the network isolation of a cluster with no route to the Internet are theirs: how the image gets in and is verified inside the enclave, patching without feeds, time and the internal PKI; here the cluster, its admission and its manifests), finops-standards (requests/limits, scaling and scheduling belong here; their cost and the split between teams sharing a node, theirs — the allocation problem in Kubernetes is real and their method solves it), platform-engineering-standards (the cluster and its operation belong here; the abstraction offered on top to the product team is theirs — if the developer writes Kubernetes YAML by hand, the platform has not done its job, and that is their decision, not this skill's), windows-server-ad-standards (the gMSA account in the directory is theirs: KDS root key, New-ADServiceAccount, PrincipalsAllowedToRetrieveManagedPassword, SPN, joining the node to the domain; its consumption from the Pod —CRD, RBAC and gmsaCredentialSpecName— belongs here), dotnet-framework-legacy-standards (whether the app can be ported to modern .NET and at what cost: that decision is theirs and overrides the decision to containerise; here only how the Windows container is run if the answer is that it is not ported) and legacy-modernization-standards (the rehost/replatform/rewrite strategy for the monolith), hpc-standards (the criteria for when batch should not run on Kubernetes and stays on Slurm are theirs; here queues and gang scheduling inside the cluster) and gpu-computing-standards/mlops-standards (the GPU, its device plugin and the training pipeline; here how the batch is queued and scheduled), webassembly-standards (Wasm is sometimes sold as a container replacement: the node runtime, admission, isolation and workload scheduling still belong here, including runwasi/containerd-shim-spin-style shims; the Wasm module, its host, its imports and its fuel and memory limits are theirs. The boundary matters because the Wasm sandbox does not replace node isolation: it only bounds what the module can ask for).
2. Default toolchain
Mandatory web verification: these versions were checked in August 2026. Before pinning versions in a real project, re-verify with WebSearch (official releases + endoflife.date). Do not pin from memory.
Never operate a minor outside upstream/managed support
Helm
4.2.x (Helm 3 EOL: fixes until 2026-09, security until 2027-02)
New projects on Helm 4; plan the migration of v2 charts
Argo CD
3.4.x (3.5 in RC: internal mTLS, commit signature verification)
Only 3 minors receive patches: keep a quarterly cadence
Flux
2.8.x
A valid alternative; pick one per organisation, not both
Kyverno
1.18.x
Admission by default (CEL + ImageValidatingPolicy); OPA/Gatekeeper only if Rego is already there
cosign
3.x (bundle format by default; v4 will remove deprecated flags)
Do not use deprecated v2 flags in new pipelines
Grype + Syft (image and SBOM), checkov/kubescape (manifests)
latest stable
Default after the Trivy supply-chain compromise (March 2026): tag poisoning of trivy-action/setup-trivy with theft of CI secrets. If you use Trivy, pin by SHA/digest, verify the signature and follow its advisories — the scanner runs in CI with access to secrets by design
Ingress NGINX is retired (no releases or patches since 2026-03-24): forbidden in new deployments; migrate to Gateway API (Envoy Gateway, Cilium, the provider's ingress).
Gateway API over Ingress for new north-south traffic.
3. Structure and conventions
Dockerfile / OCI image
Multi-stage always: build stage with the full toolchain, minimal final stage. The final artifact contains no compilers, package shells or caches.
Minimal base: distroless (gcr.io/distroless/*), chainguard or scratch for static binaries. Alpine only if you need a shell and you justify it.
Pin by digest on the base image: FROM registry/image:tag@sha256:... (the tag is documentation; the digest is the contract). Renovate/Dependabot updates the digests.
Non-root: numeric USER (USER 65532:65532), never USER app (a name does not verify the UID at admission).
One process per container; ENTRYPOINT in exec form (["binary"]), signals propagated correctly (correct PID 1 or tini).
.dockerignore mandatory; no secrets in build args or in layers (use BuildKit's --mount=type=secret).
OCI labels (org.opencontainers.image.source|revision|version) for traceability.
Canonical skeleton:
FROM golang:1.24-bookworm@sha256:<digest> AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app
FROM gcr.io/distroless/static-debian12:nonroot@sha256:<digest>
COPY --from=build /out/app /app
USER 65532:65532
ENTRYPOINT ["/app"]
K8s manifests (every prod workload)
resources.requests and limits always: realistic requests (basis for scheduling and HPA); memory limit = request (avoids surprise OOM); CPU limit optional and justified (throttling).
Probes: readinessProbe mandatory; livenessProbe only if the process can hang without dying (misused, it kills healthy pods); startupProbe for slow starts.
Explicit securityContext (pod and container): runAsNonRoot: true, readOnlyRootFilesystem: true (+ emptyDir for tmp), allowPrivilegeEscalation: false, capabilities.drop: [ALL], seccompProfile: RuntimeDefault.
Pod Security Standards restricted enforced per namespace (labels pod-security.kubernetes.io/enforce: restricted); exceptions documented per workload, never privileged at the level of an application namespace.
PDB for every Deployment/StatefulSet with >1 replica (maxUnavailable: 1 as a sane default). Without a PDB, a node drain is an incident.
Default-deny NetworkPolicy (ingress and egress) per namespace + explicit allowlist per flow. DNS (53/UDP+TCP towards kube-dns) is the only implicitly permitted egress.
topologySpreadConstraints or multi-AZ anti-affinity for replicas; priorityClassName defined.
≥2 replicas in prod; HPA on real metrics, not just CPU if the bottleneck is elsewhere.
:latest and mutable tags forbidden in image:; imagePullPolicy: Always forbidden as a patch for mutable tags.
Canonical container securityContext (the default, not the exception):
Helm for distributable/parameterisable software; Kustomize for your own environment overlays. Recommended pattern: base chart + per-environment overlays, or per-environment values in Git (values-<env>.yaml).
Charts: values.yaml with safe defaults (the chart renders safely with no overrides), values.schema.json mandatory, helm-docs for the README, SemVer versioning of the chart independent of appVersion.
No complex logic in templates: if you need more than one nested if/range, move the decision up to values or to a helper.
Charts published in an OCI registry (not legacy HTTP repos), signed with cosign.
GitOps (Argo CD / Flux)
Git is the only path to prod: every change of cluster state goes through a PR. Manual kubectl apply/edit/scale against prod is forbidden except for documented break-glass (and it is reconciled afterwards).
App-of-apps pattern (Argo) or hierarchical Kustomization (Flux); a "deploy" repo/dir separate from the app code (or a dedicated directory with CODEOWNERS).
syncPolicy.automated with prune: true and selfHeal: true in non-production environments; in prod, auto-sync only with gates (health checks + windows) or approved manual sync — an explicit decision per organisation.
Drift = alert: selfHeal or notification, never silent drift.
Promotion between environments = promotion of the same image digest through Git (image updater or automated PR), never a rebuild per environment.
Windows nodes and containers
Everything above assumes Linux. On a Windows node the canonical invariants of §3 are not degraded: they are inapplicable or false, and the expensive failure mode is a manifest that declares them, passes admission and protects nothing. Typical scenario: a .NET Framework monolith on IIS that is containerised because it cannot be ported.
Host↔image compatibility (the hard rule). The host's OS build and the base image's must match; only the revisions (4th digit) may differ from 1809 onwards. Microsoft offers Hyper-V isolation as an escape hatch for mismatched versions, but Kubernetes does not support it ("Kubernetes does not support running Windows containers with Hyper-V isolation") — in a cluster that escape hatch does not exist: the image simply does not start (0xc0370101, ContainerCannotRun → CrashLoopBackOff).
Host
Images that start on K8s (process isolation)
Windows Server 2025
ltsc2025, ltsc2022
Windows Server 2022
ltsc2022
Windows Server 2019
ltsc2019 (but WS2019 is no longer a supported node)
Kubernetes 1.36 supports Windows Server 2022 and 2025 as nodes, nothing else. WS2022 leaves mainstream on 2026-10-14 (extended until 2031-10-15) and "Containers released with Windows Server 2022 follow the same lifecycle dates" → new nodes and images go to ltsc2025; a WS2022 node today is born with two months of full support left.
Automatic label node.kubernetes.io/windows-build: WS2022 = 10.0.20348, WS2025 = 10.0.26100. With two Windows versions in the cluster it is mandatory in the nodeSelector, not optional.
Base image: there is no distroless, no scratch, nothing like it. Sizes measured from the MCR manifests (compressed in registry, Aug 2026):
Image
Size
What it ships
windows/nanoserver:ltsc2025
0.19 GB
No PowerShell, no WMI, no servicing stack, no .NET Framework. Modern .NET only
windows/servercore:ltsc2025
2.3 GB
The only one that runs .NET Framework and IIS
windows/server:ltsc2025
6.55 GB
Full API, GPU, no IIS connection limit. Only if Server Core is not enough
An IIS monolith goes to Server Core: that is a platform fact, not a choice. Start from mcr.microsoft.com/dotnet/framework/aspnet:4.8.1-windowsservercore-ltsc2025 instead of installing IIS by hand.
No latest tag on these images since 2019-04-16; the digest pinning of §3 applies just the same, and now it weighs 12-30× more on pull and on node disk.
One process per container is not met: Server Core starts the Service Control Manager and IIS runs as a service (w3wp). Do not try to force it; monitor the service, not PID 1.
USER 65532:65532 is impossible: Windows has no UID/GID, identity is a SID. The equivalent is USER ContainerUser in the Dockerfile or securityContext.windowsOptions.runAsUserName in the Pod (ContainerUser, ContainerAdministrator, NT AUTHORITY\NETWORK SERVICE…).
The default user is not non-root: verified against the config blob in MCR, servercore and windows/serverdo not set USER → the process runs as ContainerAdministrator; only nanoserver sets USER ContainerUser (ltsc2022 and ltsc2025). That is, the image the IIS monolith needs is exactly the one that starts as administrator: setting runAsUserName is mandatory, not hygiene.
A securityContext that is silently ignored — the dangerous failure. With .spec.os.name: windows, the API server rejects a Pod that sets hostPID, hostIPC, shareProcessNamespace, seLinuxOptions, seccompProfile, fsGroup, fsGroupChangePolicy, sysctls, supplementalGroups, runAsUser, runAsGroup, capabilities, readOnlyRootFilesystem, privileged, allowPrivilegeEscalation or procMount. Without .spec.os.name, the very same manifest is admitted and those fields do absolutely nothing. The canonical securityContext of §3 on a Windows node passes CI, passes admission, passes the checklist — and gives zero protection.
→ .spec.os.name is mandatory on every Pod (windows and linux alike): it is the only mechanism that turns that silence into a rejection. Kyverno gate to require it.
At the Pod level only securityContext.runAsNonRoot and securityContext.windowsOptions work. Here runAsNonRoot: true means "not ContainerAdministrator", not "UID≠0".
readOnlyRootFilesystemis not implementable on Windows ("write access is required"): do not ask for it, do not audit it, do not put it in the exception.
PSS restricted stops enforcing half of it: privilege escalation, seccomp and capabilities are Linux only controls since 1.25 (spec.os.name != windows). What is left alive: runAsNonRoot, no HostProcess, no host namespaces, no hostPath. A mixed namespace labelled restricteddoes not mean the same thing for each half — document what it actually controls on each.
There are no privileged containers on Windows; the equivalent is HostProcess (windowsOptions.hostProcess: true, stable since 1.26), which runs on the host with its privileges. Treat it like privileged: forbidden except for a signed exception for node agents.
Scheduling in a mixed cluster.
nodeSelector: kubernetes.io/os: windows on every Windows workload; .spec.os.namedoes not affect scheduling.
Taint the Windows nodes (--register-with-taints='os=windows:NoSchedule') + toleration in the Pod. Without the taint, any Linux Deployment with no selector lands on a Windows node.
A Linux DaemonSet without a nodeSelector breaks by definition: a DaemonSet goes to every node that tolerates its taints, and many agents (CNI, logs, node-exporter, runtime scanner, CSI) ship broad tolerations that cancel out the taint. Review every DaemonSet in the cluster before adding the first Windows node, not after the incident.
RuntimeClass with scheduling.nodeSelector + tolerations encapsulates the pair when there are many workloads.
The kubelet on Windows does not enforce memory or CPU limits: --kube-reserved/--system-reserved only subtract from NodeAllocatable, PIDPressure is not implemented and there is no OOM eviction. limits do not protect the node the way they do on Linux → size with headroom and alert on node memory. There is also no full kubectl exec, no pod metrics, no HPA, no ResourceQuota and no scheduler preemption with the same semantics: do not take any part of §6 for granted without checking it on Windows.
gMSA: domain identity for the Pod (what an IIS monolith almost always needs for integrated authentication against AD).
CRD GMSACredentialSpec, group windows.k8s.io, apiVersion: windows.k8s.io/v1 (v1alpha1 deprecated and non-storage), plus the two webhooks from kubernetes-sigs/windows-gmsa (v0.13.0, Apache-2.0, verified in raw): one mutating that expands the name into the full credspec, one validating that checks authorisation.
Authorisation = RBAC over the credspec: apiGroups: ["windows.k8s.io"], resources: ["gmsacredentialspecs"], verbs: ["use"], resourceNames: [<credspec>], bound to the workload's ServiceAccount. Without that binding the webhook rejects. One credspec per application, never a shared one: it is a domain identity, not a ConfigMap.
Consumption: securityContext.windowsOptions.gmsaCredentialSpecName, at Pod or container level.
The password is never in the cluster: the node retrieves it from Active Directory. Design consequence: the Windows node is a domain principal and any authorised pod on that node acts as the service account — compromise of the node is compromise of the account. A domain-joined Windows node inside a multi-tenant cluster is a risk decision, not an installation detail.
Boundary: the account in the directory belongs to windows-server-ad-standards (§1).
Coupled patching (real operational impact). The Windows base image has no servicing stack: it is not patched inside, it is rebuilt. Microsoft republishes the bases on the second Tuesday of every month ("B release") and that is "the only regular release that include new security fixes".
Mandatory monthly rebuild of every Windows image even if the app does not change, aligned with the B cycle — not "when convenient". The scheduled rebuild of §7 here has a fixed date imposed from outside.
The host is patched in the same window and at the same cadence: the node's patching calendar overrides the app's deployment calendar, the opposite of Linux.
Host and image do not need the same revision (1809+), but they do need the same build. A WS2022→WS2025 node upgrade forces rebuilding every image to ltsc2025 first, not afterwards.
When NOT to containerise Windows (honest criteria, decided before writing the Dockerfile):
Interactive installer, GUI, drivers, access to hardware or to the host registry, or local state that must survive a restart → VM. There is no shortcut.
If the real destination is modern .NET: porting to .NET 8+ and a Linux container is usually less total work than containerising the Framework, and the result does get the invariants of §3 (distroless, non-root, read-only, full PSS). That comparison is decided by dotnet-framework-legacy-standards.
If there are no Windows nodes in production already, the cost is not the image: it is a second node pool with licences, coupled monthly patching, 2-6.5 GB images, and the whole ecosystem of DaemonSets, observability and security duplicated and verified on Windows. For 1-3 applications it rarely pays off.
An app deployed twice a year gains nothing from Kubernetes: what you gain is scaling and frequent deployment. If you do not need them, the VM is cheaper and simpler.
Containerising Windows is valid as an intermediate step with an exit date, not as a permanent destination. If there is no date, it is an expensive VM with more moving parts.
4. Quality and testing (CI gates)
Minimum pipeline that breaks the build:
Lint: hadolint (Dockerfile), yamllint, helm lint + render (helm template with no errors), kustomize build of every overlay.
Schema validation: kubeconform (with CRD schemas) over the rendered YAML of all environments.
Policies in CI (shift-left): kyverno apply / conftest with the same policies as the cluster's admission — what breaks at admission must break earlier in CI.
Chart tests: helm unittest for template logic; chart-testing (ct lint/ct install) against kind/k3d in chart PRs.
Scanning: Grype + Syft over the image (CVEs and SBOM, gate on CRITICAL/HIGH with documented triage) and checkov/kubescape over the manifests — not Trivy by default (§2), and if it is used, pinned by digest and with its advisories followed. Secret scanning: the scanner and its licence are set by secrets-management-standards (as of Aug 2026 gitleaks is feature complete and its GitHub action requires a commercial licence for organisations: verify before pinning it).
Visible diff in the PR: render of the resulting YAML (argocd diff / flux diff / helm diff) as a comment — what is going to be applied is reviewed, not just the template.
5. Security
Supply chain
Signing with cosign (keyless, CI's OIDC) of every image and chart that reaches prod; verification at admission (Kyverno ImageValidatingPolicy/verifyImages): an image without a valid signature does not run.
SBOM (syft, SPDX or CycloneDX format) generated at build, attached as an attestation (cosign attest), not as a loose file.
SLSA provenance (builder provenance attestation) verified where the registry/CI supports it.
Private registry with pull-through cache; direct pulls from Docker Hub forbidden in prod (rate limits + supply chain).
Renovate/Dependabot for bases and images: updating a digest is a PR, not a manual event.
Runtime and cluster
Admission with Kyverno in Enforce mode for the baseline (non-root, no privileged, digests, allowed registries, requests/limits, probes); Audit only as an introduction phase, with a date for moving to Enforce.
Minimal RBAC: no cluster-admin for humans or for app ServiceAccounts; automountServiceAccountToken: false unless needed.
Secrets never in the clear in Git: External Secrets Operator against Vault/Secrets Manager (preferred) or SOPS+age if there is no manager. Secret in plain YAML or in Helm values without encryption is forbidden.
Secrets encryption at rest in etcd (KMS provider) and east-west mTLS (mesh or Cilium) where the risk profile calls for it.
6. Operability
Non-negotiable observability: Prometheus metrics (/metrics), structured JSON logs to stdout, OTLP traces. Without telemetry, there is no prod deployment.
Real health endpoints: readiness reflects critical dependencies (with judgement: do not take the pod down for a degraded dependency that has a fallback).
Safe deployment: rolling update with maxUnavailable: 0 as the default; canary/blue-green (Argo Rollouts or Flagger) for critical services, with automatic analysis over metrics.
Tested rollback: helm rollback/Git revert rehearsed in staging; a backwards-incompatible data migration blocks the rollback — use expand/contract.
terminationGracePeriodSeconds matched to the app's real drain; the app handles SIGTERM.
Alerts on symptoms (golden signals) + GitOps state (app OutOfSync/Degraded, failed reconcile), not on every restart.
Batch queues and gang scheduling
All of §6 assumes long-lived services. For batch (distributed training, simulation, bulk ETL) the cluster does not ship what is needed and it has to be added explicitly.
What the default scheduler does not do.kube-scheduler"selects an optimal node to run newly created or not yet scheduled (unscheduled) pods" in two phases, filtering and scoring, and assigns Pod by Pod. There is no job queue, no service priority between batches, no group semantics: neither the scheduler docs nor the Job docs mention gang, queuing or all-or-nothing.
What that breaks. A job that needs its N pods at the same time (synchronous training, MPI, anything with pod-to-pod communication) can end up with N-1 pods running and occupying resources, waiting indefinitely for the last one. With two such jobs competing, each blocks the other: resource deadlock. Verbatim from the Kueue docs: "a pair of such jobs may deadlock if the physical availability of resources do not match the configured quotas in Kueue. The same pair of jobs could run to completion if their pods were scheduled sequentially." Nobody detects it and nothing breaks: the signal is low utilisation with eternally Pending pods and a cluster that "is full". Outside a real load test it does not show up; in production, it does.
ResourceQuota/LimitRange are not the solution and mistaking them for it is the usual error. ResourceQuota"provides constraints that limit aggregate resource consumption per namespace", and on violation "the control plane rejects that request with HTTP status code 403 Forbidden": that is rejection at admission, not queuing — no service order, no retry, no "it gets in when there is room". It also acts object by object, so nothing stops it admitting N-1 pods of the job and rejecting the last one. LimitRange only sets defaults and min/max per object. They are consumption guardrails; they are not a scheduler.
schedulingGates (Pod Scheduling Readiness, GA in 1.30) and Job.spec.suspend are the hooks on which a queue manager is built — they hold individual pods or objects so an external integrator can decide. They provide no group semantics by themselves.
The two real options (verified Aug 2026; both Apache-2.0 read in raw):
Kueue
Volcano
What it is
Job-level queue manager on top of the default scheduler
Full batch scheduler, replacing/added to the default one
Version
v0.19.0 (2026-07-22)
v1.15.1 (2026-07-30)
Governance
kubernetes-sigs, official SIG Scheduling subproject
waitForPodsReady in the controller's Configuration
gang plugin over PodGroup.minMember/minResources
Requirement
K8s ≥ 1.29
—
Kueue suspends and queues the Workload until there is quota; waitForPodsReady (30 min timeout by default, recoveryTimeout, requeuingStrategy with backoff) evicts and requeues the job if not all its pods start. With blockAdmission: true it admits sequentially to break the deadlock — at the cost of serialising starts even when there is plenty of room. It is a mitigation, not strict gang scheduling.
Volcano does real gang: minMember — "if there's not enough resources to start all tasks, the scheduler will not start anyone" — plus fair-share between Queues and scheduling policies (binpack, NUMA, topology). The price: a second scheduler in the cluster and a Job API still on v1alpha1 despite years in incubation. Weigh it up.
Criteria: if what is missing is queues, per-team quotas and job admission, Kueue (you keep a single scheduler and it fits Job, JobSet and the training operators). If you need strict gang and scheduling policies, Volcano. One per organisation, not both: two schedulers deciding over the same node pool is an incident waiting to happen; if they coexist, partition by taints and be explicit about which nodes each one manages.
JobSet (jobset.x-k8s.io/v1alpha2, v0.12.0, sig-apps) groups several Jobs as a unit for HPC/ML. Still alpha: it complements Kueue, it does not replace it.
Forbidden to simulate gang by hand: initContainers waiting for their peers, creative podAffinity or startup sleeps are the same deadlock with more steps and no observability.
When the right answer is not to use Kubernetes. If the batch is classic HPC —tight MPI, low-latency interconnect, whole-node reservation, accounting by CPU-hours, users submitting with sbatch— the answer is usually Slurm, not a Kubernetes cluster with two operators on top. Those criteria belong to hpc-standards and are consulted there before building anything here. The practical boundary: if what you already have lives in Kubernetes and all that is missing are queues, Kueue/Volcano; if what you have is a computing centre, do not reimplement it in YAML.
7. Sustainability and prohibitions
Upgrade cadence: K8s publishes 3 minors/year and supports N-2 → plan at least 2 minor upgrades per year; never more than one minor behind the managed provider's support. Before upgrading: read the deprecations (kubectl api-resources, Pluto for removed APIs), update charts/operators, rehearse in staging.
Argo CD/Flux/Kyverno: stay within the 3 supported minors (quarterly review cadence).
Refresh base images even if the app does not change (scheduled weekly/monthly rebuild: the CVEs arrive on their own).
FORBIDDEN (automatic gate wherever possible):
:latest or mutable tags in prod; images without a digest.
Manual kubectl apply/edit/scale/port-forward against prod outside documented break-glass.
Root containers, privileged, hostNetwork/hostPID/hostPath without a signed exception.
Secrets in the clear in Git, in values, in the manifest's env or in logs.
Workloads without requests/limits, without a readiness probe or without a PDB (multi-replica) in prod.
Application namespaces without a default-deny NetworkPolicy or PSS restricted.
An unsigned image or one without an SBOM on the path to prod.
Ingress NGINX in new deployments (retired project, no security patches).
Helm 2/3 in new projects; charts without values.schema.json.
Tolerated drift: resources in the cluster that do not exist in Git.
A mixed Linux/Windows cluster with Pods lacking .spec.os.name, lacking a kubernetes.io/osnodeSelector or with untainted Windows nodes; a Linux DaemonSet without a selector after adding the first Windows node.
Taking the protection of a Linux securityContext (seccomp, capabilities, readOnlyRootFilesystem, allowPrivilegeEscalation) at face value on a Windows workload: it does not protect, it is ignored.
A Windows image without a monthly rebuild aligned with the "B release"; a base image whose build does not match the node's.
Gang scheduling simulated with initContainers, affinities or waits; two batch schedulers over the same node pool without partitioning.
Quick review checklist (every workload PR to prod)
Image by digest, signed, with an SBOM; distroless/minimal base and numeric non-root.
Full restricted securityContext; namespace with PSS restricted + default-deny NetworkPolicy.
No secrets in the clear; dedicated ServiceAccount with minimal RBAC and the token not mounted if unused.
Render validated (kubeconform) + admission policies passed in CI; YAML diff visible in the PR.
Metrics/logs/traces exposed; SIGTERM handled; rollback rehearsed or an expand/contract plan if there is a migration.
If Windows: .spec.os.name: windows, nodeSelector+toleration, image build equal to the node's (node.kubernetes.io/windows-build), runAsUserName set, no Linux securityContext fields, and a monthly rebuild planned.
If multi-pod batch: queue manager (Kueue or Volcano) declared and all-or-nothing semantics configured — not ResourceQuota as a substitute.
8. Mandatory web verification
Before pinning any version, API (apiVersion), flag or feature in deliverables:
WebSearch/WebFetch of the component's official release and endoflife.date (K8s, Helm, Argo CD/Flux, Kyverno, cosign) — the ecosystem rotates every quarter and this document ages.
Verify the K8s API deprecations of the target version jump (official release notes + Pluto).
Cosign: confirm whether v4 has shipped (it removes v3's deprecated flags) before writing signing pipelines.
Windows: re-verify on MS Learn (version-compatibility, container-base-images, update-containers) the host↔image matrix and the lifecycle before choosing a base — it is where the most stale information circulates. Check on kubernetes.io/docs/concepts/windows/intro which Windows Server versions are supported nodes on your K8s minor (as of 1.36: only 2022 and 2025) and whether the list of unsupported securityContext fields has changed. WS2022 leaves mainstream on 2026-10-14: from that date this table no longer describes a viable option for new nodes. The image sizes were measured from the MCR manifests in Aug 2026 and change with every "B release": measure them again (/v2/<img>/manifests/<tag> → sum of layers[].size), do not quote them from here.
Batch: re-verify the version and maturity of Kueue (is it still on v1beta2 or is there a v1 GA?) and of Volcano (is it still Incubating in the CNCF or has it graduated? is its Job still on v1alpha1?), and the status of JobSet (alpha as of Aug 2026). Cross-check the project's official site against api.github.com, not against the releases feed.
If you cannot verify, say so explicitly in the deliverable instead of assuming.
Declared gaps (Aug 2026), to be verified before using them as data:
The on-disk (uncompressed) sizes of the Windows images were not measured: the figures in the table are the registry's compressed sizes. On node disk they are significantly larger.
Kueue's official installation page (kueue.sigs.k8s.io/docs/installation/) could not be fetched: v0.19.0 is backed by api.github.com + the Atom feed + the repo source at the tag, but without confirmation from the project's website.
Of Kueue's CRDs, the manifests for ClusterQueue, AdmissionCheck, Topology and Workload were read; LocalQueue and ResourceFlavor are assumed to follow the same v1beta1 deprecated / v1beta2 storage pattern by extrapolation, not by direct reading.
For the Volcano Job, the group, kind and v1alpha1 version were confirmed, but not the served/storage booleans of the manifest. The chain Job.minAvailable → PodGroup.minMember is reasoned inference: the CRD does not document that field.
The CNCF TAG that Volcano belongs to was not verified against a current primary source.
The v0.13.0 version of windows-gmsa comes from api.github.com; the project does not publish its own website to cross-check it against.
If the web contradicts this document, the web wins — flag the discrepancy.
1---2name: kubernetes-standards3description: Kubernetes and container standards (staff/principal level). Use when writing or reviewing Dockerfiles/Containerfiles, OCI images, Kubernetes manifests (*.yaml with apiVersion/kind), Helm charts (Chart.yaml, values.yaml, templates/), Kustomize overlays (kustomization.yaml), GitOps configs (Argo CD Application, Flux Kustomization/HelmRelease), admission policies (Kyverno, OPA/Gatekeeper), image signing (cosign) or SBOM tooling. Also for Windows nodes and Windows containers (Server Core, Nano Server, ltsc2022/ltsc2025 base images, host/image version compatibility, runAsUserName, HostProcess, gMSA and GMSACredentialSpec) and for batch scheduling on Kubernetes (job queues, gang scheduling, Kueue, Volcano, ResourceQuota).4---56# Kubernetes and container standards78## 1. Scope and triggers910Applies when creating or reviewing: `Dockerfile`/`Containerfile`, K8s manifests (`*.yaml` with `apiVersion`/`kind`), Helm charts (`Chart.yaml`, `templates/`), `kustomization.yaml`, Argo CD/Flux resources, Kyverno/OPA policies, pipelines that build/sign/deploy images. Does not apply to local development Compose (use proportionate judgement).1112**Not applicable**: see `iac-standards` (provisioning of the cluster and the infrastructure with Terraform/OpenTofu and Ansible; the boundary is decided there: **cluster and platform with TF/Tofu, workloads via GitOps**), `aws-standards`/`azure-standards`/`gcp-standards` (the managed control plane —EKS/AKS/GKE—, its integration with IAM/VPC and its cost; here what runs inside), `container-runtime-security-standards` (what happens **after the Pod starts**: seccomp, runtime choice and pinning, container escape, runtime detection with Falco/Tetragon/eBPF, drift and node forensics; here admission, Kyverno policies, Pod Security Standards, declarative `securityContext` and signature verified at admission), `selinux-standards` (the container's MAC: `container_t`, MCS, `:z`/`:Z`, `udica`, AppArmor profiles), `networking-standards` (underlying physical network, VLAN, BGP and MTU; here Services, Gateway API and NetworkPolicy), `cicd-standards` (the pipeline that builds and signs the image and triggers the deployment; here the resulting manifest and the verification at admission), `observability-standards` (Prometheus, OTel Collector and their rules; here only probes, resources and the `ServiceMonitor`), `sre-practice-standards` (SLOs, capacity and on-call), `appsec-standards` (application code), `vulnerability-management-standards` (triage and SLA of the CVEs Grype/Syft report), `cryptography-pki-standards` (signing key management and internal PKI; here only the use of cosign and cert-manager), `identity-access-management-standards` (IdP and federation; here cluster RBAC and ServiceAccounts), `homelab-standards` and `onprem-standards` (single-node k3s/Talos and the hardware/OS underneath), `air-gapped-standards` (**the internal registry/mirror and the network isolation of a cluster with no route to the Internet are theirs**: how the image gets in and is verified inside the enclave, patching without feeds, time and the internal PKI; here the cluster, its admission and its manifests), `finops-standards` (`requests`/`limits`, scaling and scheduling belong here; **their cost and the split between teams sharing a node, theirs** — the allocation problem in Kubernetes is real and their method solves it), `platform-engineering-standards` (**the cluster and its operation belong here**; **the abstraction offered on top to the product team is theirs** — if the developer writes Kubernetes YAML by hand, the platform has not done its job, and that is their decision, not this skill's), `windows-server-ad-standards` (**the gMSA account in the directory is theirs**: KDS root key, `New-ADServiceAccount`, `PrincipalsAllowedToRetrieveManagedPassword`, SPN, joining the node to the domain; **its consumption from the Pod —CRD, RBAC and `gmsaCredentialSpecName`— belongs here**), `dotnet-framework-legacy-standards` (whether the app can be ported to modern .NET and at what cost: that decision is theirs and **overrides the decision to containerise**; here only how the Windows container is run if the answer is that it is not ported) and `legacy-modernization-standards` (the rehost/replatform/rewrite strategy for the monolith), `hpc-standards` (**the criteria for when batch should not run on Kubernetes and stays on Slurm are theirs**; here queues and gang scheduling *inside* the cluster) and `gpu-computing-standards`/`mlops-standards` (the GPU, its device plugin and the training pipeline; here how the batch is queued and scheduled), `webassembly-standards` (Wasm is sometimes sold as a container replacement: **the node runtime, admission, isolation and workload scheduling still belong here**, including runwasi/`containerd-shim-spin`-style *shims*; **the Wasm module, its host, its imports and its fuel and memory limits are theirs**. The boundary matters because **the Wasm sandbox does not replace node isolation**: it only bounds what the module can ask for).1314## 2. Default toolchain1516> **Mandatory web verification**: these versions were checked in **August 2026**. Before pinning versions in a real project, re-verify with WebSearch (official releases + endoflife.date). Do not pin from memory.1718| Tool | Stable line (2026-08) | Criteria |19|---|---|---|20| Kubernetes | **1.36.x** (1.37 ships 2026-08-26); supported N-2: 1.34–1.36 | Never operate a minor outside upstream/managed support |21| Helm | **4.2.x** (Helm 3 EOL: fixes until 2026-09, security until 2027-02) | New projects on Helm 4; plan the migration of v2 charts |22| Argo CD | **3.4.x** (3.5 in RC: internal mTLS, commit signature verification) | Only 3 minors receive patches: keep a quarterly cadence |23| Flux | **2.8.x** | A valid alternative; pick one per organisation, not both |24| Kyverno | **1.18.x** | Admission by default (CEL + `ImageValidatingPolicy`); OPA/Gatekeeper only if Rego is already there |25| cosign | **3.x** (bundle format by default; v4 will remove deprecated flags) | Do not use deprecated v2 flags in new pipelines |26| **Grype + Syft** (image and SBOM), **checkov**/**kubescape** (manifests) | latest stable | Default after the **Trivy supply-chain compromise (March 2026)**: tag poisoning of `trivy-action`/`setup-trivy` with theft of CI secrets. If you use Trivy, **pin by SHA/digest**, verify the signature and follow its advisories — the scanner runs in CI with access to secrets by design |2728- **Ingress NGINX is retired** (no releases or patches since 2026-03-24): forbidden in new deployments; migrate to Gateway API (Envoy Gateway, Cilium, the provider's ingress).29- Gateway API over Ingress for new north-south traffic.3031## 3. Structure and conventions3233### Dockerfile / OCI image34- **Multi-stage always**: build stage with the full toolchain, minimal final stage. The final artifact contains no compilers, package shells or caches.35- **Minimal base**: distroless (`gcr.io/distroless/*`), chainguard or `scratch` for static binaries. Alpine only if you need a shell and you justify it.36- **Pin by digest** on the base image: `FROM registry/image:tag@sha256:...` (the tag is documentation; the digest is the contract). Renovate/Dependabot updates the digests.37- **Non-root**: numeric `USER` (`USER 65532:65532`), never `USER app` (a name does not verify the UID at admission).38- One process per container; `ENTRYPOINT` in exec form (`["binary"]`), signals propagated correctly (correct PID 1 or tini).39- `.dockerignore` mandatory; no secrets in build args or in layers (use BuildKit's `--mount=type=secret`).40- OCI labels (`org.opencontainers.image.source|revision|version`) for traceability.4142Canonical skeleton:4344```dockerfile45FROM golang:1.24-bookworm@sha256:<digest> AS build46WORKDIR /src47COPY go.mod go.sum ./48RUN go mod download49COPY . .50RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app5152FROM gcr.io/distroless/static-debian12:nonroot@sha256:<digest>53COPY --from=build /out/app /app54USER 65532:6553255ENTRYPOINT ["/app"]56```5758### K8s manifests (every prod workload)59- **`resources.requests` and `limits` always**: realistic requests (basis for scheduling and HPA); memory limit = request (avoids surprise OOM); CPU limit optional and justified (throttling).60- **Probes**: `readinessProbe` mandatory; `livenessProbe` only if the process can hang without dying (misused, it kills healthy pods); `startupProbe` for slow starts.61- **Explicit `securityContext`** (pod and container): `runAsNonRoot: true`, `readOnlyRootFilesystem: true` (+ `emptyDir` for tmp), `allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, `seccompProfile: RuntimeDefault`.62- **Pod Security Standards `restricted`** enforced per namespace (labels `pod-security.kubernetes.io/enforce: restricted`); exceptions documented per workload, never `privileged` at the level of an application namespace.63- **PDB** for every Deployment/StatefulSet with >1 replica (`maxUnavailable: 1` as a sane default). Without a PDB, a node drain is an incident.64- **Default-deny NetworkPolicy** (ingress and egress) per namespace + explicit allowlist per flow. DNS (53/UDP+TCP towards kube-dns) is the only implicitly permitted egress.65- `topologySpreadConstraints` or multi-AZ anti-affinity for replicas; `priorityClassName` defined.66- ≥2 replicas in prod; HPA on real metrics, not just CPU if the bottleneck is elsewhere.67- `:latest` and mutable tags forbidden in `image:`; `imagePullPolicy: Always` forbidden as a patch for mutable tags.6869Canonical container `securityContext` (the default, not the exception):7071```yaml72securityContext:73 runAsNonRoot: true74 runAsUser: 6553275 readOnlyRootFilesystem: true76 allowPrivilegeEscalation: false77 capabilities: { drop: ["ALL"] }78 seccompProfile: { type: RuntimeDefault }79```8081Base default-deny NetworkPolicy per namespace (ingress+egress, DNS allowed):8283```yaml84apiVersion: networking.k8s.io/v185kind: NetworkPolicy86metadata: { name: default-deny }87spec:88 podSelector: {}89 policyTypes: [Ingress, Egress]90 egress:91 - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]92 ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]93```9495### Helm / Kustomize96- **Helm** for distributable/parameterisable software; **Kustomize** for your own environment overlays. Recommended pattern: base chart + per-environment overlays, or per-environment values in Git (`values-<env>.yaml`).97- Charts: `values.yaml` with safe defaults (the chart renders safely with no overrides), `values.schema.json` mandatory, `helm-docs` for the README, SemVer versioning of the chart independent of `appVersion`.98- No complex logic in templates: if you need more than one nested `if/range`, move the decision up to values or to a helper.99- Charts published in an **OCI registry** (not legacy HTTP repos), signed with cosign.100101### GitOps (Argo CD / Flux)102- **Git is the only path to prod**: every change of cluster state goes through a PR. Manual `kubectl apply`/`edit`/`scale` against prod is forbidden except for documented break-glass (and it is reconciled afterwards).103- **App-of-apps** pattern (Argo) or hierarchical `Kustomization` (Flux); a "deploy" repo/dir separate from the app code (or a dedicated directory with CODEOWNERS).104- `syncPolicy.automated` with `prune: true` and `selfHeal: true` in non-production environments; in prod, auto-sync only with gates (health checks + windows) or approved manual sync — an explicit decision per organisation.105- Drift = alert: `selfHeal` or notification, never silent drift.106- Promotion between environments = promotion of the **same image digest** through Git (image updater or automated PR), never a rebuild per environment.107108### Windows nodes and containers109110Everything above assumes Linux. On a Windows node **the canonical invariants of §3 are not degraded: they are inapplicable or false**, and the expensive failure mode is a manifest that declares them, passes admission and protects nothing. Typical scenario: a .NET Framework monolith on IIS that is containerised because it cannot be ported.111112**Host↔image compatibility (the hard rule).** The host's OS build and the base image's **must match**; only the revisions (4th digit) may differ from 1809 onwards. Microsoft offers Hyper-V isolation as an escape hatch for mismatched versions, but **Kubernetes does not support it** ("Kubernetes does not support running Windows containers with Hyper-V isolation") — in a cluster that escape hatch **does not exist**: the image simply does not start (`0xc0370101`, `ContainerCannotRun` → `CrashLoopBackOff`).113114| Host | Images that start on K8s (process isolation) |115|---|---|116| Windows Server 2025 | ltsc2025, ltsc2022 |117| Windows Server 2022 | ltsc2022 |118| Windows Server 2019 | ltsc2019 (but WS2019 is no longer a supported node) |119120- **Kubernetes 1.36 supports Windows Server 2022 and 2025 as nodes, nothing else.** WS2022 leaves mainstream on **2026-10-14** (extended until 2031-10-15) and *"Containers released with Windows Server 2022 follow the same lifecycle dates"* → new nodes and images go to **ltsc2025**; a WS2022 node today is born with two months of full support left.121- Automatic label `node.kubernetes.io/windows-build`: **WS2022 = `10.0.20348`**, **WS2025 = `10.0.26100`**. With two Windows versions in the cluster it is mandatory in the `nodeSelector`, not optional.122123**Base image: there is no distroless, no `scratch`, nothing like it.** Sizes measured from the MCR manifests (compressed in registry, Aug 2026):124125| Image | Size | What it ships |126|---|---|---|127| `windows/nanoserver:ltsc2025` | **0.19 GB** | No PowerShell, no WMI, **no servicing stack, no .NET Framework**. Modern .NET only |128| `windows/servercore:ltsc2025` | **2.3 GB** | The **only** one that runs .NET Framework and IIS |129| `windows/server:ltsc2025` | **6.55 GB** | Full API, GPU, no IIS connection limit. Only if Server Core is not enough |130131- An IIS monolith **goes to Server Core**: that is a platform fact, not a choice. Start from `mcr.microsoft.com/dotnet/framework/aspnet:4.8.1-windowsservercore-ltsc2025` instead of installing IIS by hand.132- No `latest` tag on these images since 2019-04-16; the digest pinning of §3 applies just the same, and now it weighs 12-30× more on pull and on node disk.133- **One process per container is not met**: Server Core starts the Service Control Manager and IIS runs as a service (`w3wp`). Do not try to force it; monitor the service, not PID 1.134- **`USER 65532:65532` is impossible**: Windows has no UID/GID, identity is a SID. The equivalent is `USER ContainerUser` in the Dockerfile or `securityContext.windowsOptions.runAsUserName` in the Pod (`ContainerUser`, `ContainerAdministrator`, `NT AUTHORITY\NETWORK SERVICE`…).135- **The default user is not non-root**: verified against the config blob in MCR, `servercore` and `windows/server` **do not set `USER`** → the process runs as `ContainerAdministrator`; only `nanoserver` sets `USER ContainerUser` (ltsc2022 and ltsc2025). That is, **the image the IIS monolith needs is exactly the one that starts as administrator**: setting `runAsUserName` is mandatory, not hygiene.136137**A `securityContext` that is silently ignored — the dangerous failure.** With `.spec.os.name: windows`, the API server **rejects** a Pod that sets `hostPID`, `hostIPC`, `shareProcessNamespace`, `seLinuxOptions`, `seccompProfile`, `fsGroup`, `fsGroupChangePolicy`, `sysctls`, `supplementalGroups`, `runAsUser`, `runAsGroup`, `capabilities`, `readOnlyRootFilesystem`, `privileged`, `allowPrivilegeEscalation` or `procMount`. **Without `.spec.os.name`, the very same manifest is admitted and those fields do absolutely nothing.** The canonical `securityContext` of §3 on a Windows node passes CI, passes admission, passes the checklist — and gives zero protection.138139- → **`.spec.os.name` is mandatory on every Pod** (`windows` and `linux` alike): it is the only mechanism that turns that silence into a rejection. Kyverno gate to require it.140- At the Pod level only `securityContext.runAsNonRoot` and `securityContext.windowsOptions` work. Here `runAsNonRoot: true` means "not `ContainerAdministrator`", not "UID≠0".141- `readOnlyRootFilesystem` **is not implementable** on Windows ("write access is required"): do not ask for it, do not audit it, do not put it in the exception.142- **PSS `restricted` stops enforcing half of it**: privilege escalation, seccomp and capabilities are *Linux only* controls since 1.25 (`spec.os.name != windows`). What is left alive: `runAsNonRoot`, no HostProcess, no host namespaces, no hostPath. A mixed namespace labelled `restricted` **does not mean the same thing for each half** — document what it actually controls on each.143- **There are no privileged containers on Windows**; the equivalent is **HostProcess** (`windowsOptions.hostProcess: true`, stable since 1.26), which runs on the host with its privileges. Treat it like `privileged`: forbidden except for a signed exception for node agents.144145**Scheduling in a mixed cluster.**146- `nodeSelector: kubernetes.io/os: windows` on every Windows workload; `.spec.os.name` **does not affect scheduling**.147- **Taint the Windows nodes** (`--register-with-taints='os=windows:NoSchedule'`) + toleration in the Pod. Without the taint, any Linux Deployment with no selector lands on a Windows node.148- **A Linux DaemonSet without a `nodeSelector` breaks by definition**: a DaemonSet goes to *every* node that tolerates its taints, and many agents (CNI, logs, node-exporter, runtime scanner, CSI) ship broad tolerations that cancel out the taint. **Review every DaemonSet in the cluster before adding the first Windows node**, not after the incident.149- `RuntimeClass` with `scheduling.nodeSelector` + `tolerations` encapsulates the pair when there are many workloads.150- The kubelet on Windows **does not enforce memory or CPU limits**: `--kube-reserved`/`--system-reserved` only subtract from `NodeAllocatable`, `PIDPressure` is not implemented and there is no OOM eviction. `limits` do not protect the node the way they do on Linux → size with headroom and alert on **node** memory. There is also no full `kubectl exec`, no pod metrics, no HPA, no ResourceQuota and no scheduler preemption with the same semantics: do not take any part of §6 for granted without checking it on Windows.151152**gMSA: domain identity for the Pod** (what an IIS monolith almost always needs for integrated authentication against AD).153- CRD `GMSACredentialSpec`, group `windows.k8s.io`, **`apiVersion: windows.k8s.io/v1`** (`v1alpha1` deprecated and non-storage), plus the two webhooks from `kubernetes-sigs/windows-gmsa` (**v0.13.0**, Apache-2.0, verified in raw): one *mutating* that expands the name into the full credspec, one *validating* that checks authorisation.154- Authorisation = RBAC over the credspec: `apiGroups: ["windows.k8s.io"]`, `resources: ["gmsacredentialspecs"]`, `verbs: ["use"]`, `resourceNames: [<credspec>]`, bound to the workload's ServiceAccount. Without that binding the webhook rejects. **One credspec per application**, never a shared one: it is a domain identity, not a ConfigMap.155- Consumption: `securityContext.windowsOptions.gmsaCredentialSpecName`, at Pod or container level.156- **The password is never in the cluster**: the **node** retrieves it from Active Directory. Design consequence: the Windows node is a domain principal and **any authorised pod on that node acts as the service account** — compromise of the node is compromise of the account. A domain-joined Windows node inside a multi-tenant cluster is a risk decision, not an installation detail.157- Boundary: the account in the directory belongs to `windows-server-ad-standards` (§1).158159**Coupled patching (real operational impact).** The Windows base image **has no servicing stack**: it is not patched inside, it is **rebuilt**. Microsoft republishes the bases on the **second Tuesday of every month** ("B release") and that is *"the only regular release that include new security fixes"*.160- **Mandatory monthly rebuild** of every Windows image even if the app does not change, aligned with the B cycle — not "when convenient". The scheduled rebuild of §7 here has a fixed date imposed from outside.161- The host is patched in the same window and at the same cadence: **the node's patching calendar overrides the app's deployment calendar**, the opposite of Linux.162- Host and image do not need the same revision (1809+), but they do need the same build. A WS2022→WS2025 node upgrade forces rebuilding **every** image to ltsc2025 first, not afterwards.163164**When NOT to containerise Windows** (honest criteria, decided before writing the Dockerfile):165- Interactive installer, GUI, drivers, access to hardware or to the host registry, or local state that must survive a restart → **VM**. There is no shortcut.166- If the real destination is modern .NET: **porting to .NET 8+ and a Linux container is usually less total work** than containerising the Framework, and the result does get the invariants of §3 (distroless, non-root, read-only, full PSS). That comparison is decided by `dotnet-framework-legacy-standards`.167- If there are no Windows nodes in production already, the cost is not the image: it is a **second node pool** with licences, coupled monthly patching, 2-6.5 GB images, and the whole ecosystem of DaemonSets, observability and security duplicated and verified on Windows. For 1-3 applications it rarely pays off.168- An app deployed twice a year gains nothing from Kubernetes: what you gain is scaling and frequent deployment. If you do not need them, the VM is cheaper and simpler.169- Containerising Windows is valid as an **intermediate step with an exit date**, not as a permanent destination. If there is no date, it is an expensive VM with more moving parts.170171## 4. Quality and testing (CI gates)172173Minimum pipeline that **breaks the build**:1741. **Lint**: `hadolint` (Dockerfile), `yamllint`, `helm lint` + render (`helm template` with no errors), `kustomize build` of every overlay.1752. **Schema validation**: `kubeconform` (with CRD schemas) over the rendered YAML of **all** environments.1763. **Policies in CI (shift-left)**: `kyverno apply` / `conftest` with the same policies as the cluster's admission — what breaks at admission must break earlier in CI.1774. **Chart tests**: `helm unittest` for template logic; `chart-testing` (`ct lint`/`ct install`) against kind/k3d in chart PRs.1785. **Scanning**: **Grype + Syft** over the image (CVEs and SBOM, gate on CRITICAL/HIGH with documented triage) and **checkov**/**kubescape** over the manifests — not Trivy by default (§2), and if it is used, pinned by digest and with its advisories followed. Secret scanning: the scanner and its licence are set by `secrets-management-standards` (as of Aug 2026 `gitleaks` is *feature complete* and its GitHub action requires a commercial licence for organisations: verify before pinning it).1796. **Visible diff in the PR**: render of the resulting YAML (argocd diff / flux diff / helm diff) as a comment — what is going to be applied is reviewed, not just the template.180181## 5. Security182183### Supply chain184- **Signing with cosign (keyless, CI's OIDC)** of every image and chart that reaches prod; **verification at admission** (Kyverno `ImageValidatingPolicy`/verifyImages): an image without a valid signature does not run.185- **SBOM** (syft, SPDX or CycloneDX format) generated at build, attached as an attestation (`cosign attest`), not as a loose file.186- SLSA provenance (builder provenance attestation) verified where the registry/CI supports it.187- Private registry with pull-through cache; direct pulls from Docker Hub forbidden in prod (rate limits + supply chain).188- Renovate/Dependabot for bases and images: updating a digest is a PR, not a manual event.189190### Runtime and cluster191- Admission with Kyverno in `Enforce` mode for the baseline (non-root, no privileged, digests, allowed registries, requests/limits, probes); `Audit` only as an introduction phase, with a date for moving to Enforce.192- Minimal RBAC: no `cluster-admin` for humans or for app ServiceAccounts; `automountServiceAccountToken: false` unless needed.193- **Secrets never in the clear in Git**: External Secrets Operator against Vault/Secrets Manager (preferred) or SOPS+age if there is no manager. `Secret` in plain YAML or in Helm values without encryption is forbidden.194- Secrets encryption at rest in etcd (KMS provider) and east-west mTLS (mesh or Cilium) where the risk profile calls for it.195196## 6. Operability197198- **Non-negotiable observability**: Prometheus metrics (`/metrics`), structured JSON logs to stdout, OTLP traces. Without telemetry, there is no prod deployment.199- Real health endpoints: readiness reflects critical dependencies (with judgement: do not take the pod down for a degraded dependency that has a fallback).200- **Safe deployment**: rolling update with `maxUnavailable: 0` as the default; canary/blue-green (Argo Rollouts or Flagger) for critical services, with automatic analysis over metrics.201- **Tested rollback**: `helm rollback`/Git revert rehearsed in staging; a backwards-incompatible data migration blocks the rollback — use expand/contract.202- `terminationGracePeriodSeconds` matched to the app's real drain; the app handles SIGTERM.203- Alerts on symptoms (golden signals) + GitOps state (app OutOfSync/Degraded, failed reconcile), not on every restart.204205### Batch queues and gang scheduling206207All of §6 assumes long-lived services. For batch (distributed training, simulation, bulk ETL) the cluster **does not ship what is needed** and it has to be added explicitly.208209**What the default scheduler does not do.** `kube-scheduler` *"selects an optimal node to run newly created or not yet scheduled (unscheduled) pods"* in two phases, *filtering* and *scoring*, and assigns **Pod by Pod**. There is no job queue, no service priority between batches, no group semantics: neither the scheduler docs nor the `Job` docs mention gang, queuing or all-or-nothing.210211**What that breaks.** A job that needs its N pods **at the same time** (synchronous training, MPI, anything with pod-to-pod communication) can end up with N-1 pods running and occupying resources, waiting indefinitely for the last one. With two such jobs competing, each blocks the other: **resource deadlock**. Verbatim from the Kueue docs: *"a pair of such jobs may deadlock if the physical availability of resources do not match the configured quotas in Kueue. The same pair of jobs could run to completion if their pods were scheduled sequentially."* Nobody detects it and nothing breaks: the signal is low utilisation with eternally `Pending` pods and a cluster that "is full". Outside a real load test it does not show up; in production, it does.212213**`ResourceQuota`/`LimitRange` are not the solution** and mistaking them for it is the usual error. `ResourceQuota` *"provides constraints that limit aggregate resource consumption per namespace"*, and on violation *"the control plane rejects that request with HTTP status code `403 Forbidden`"*: that is **rejection at admission, not queuing** — no service order, no retry, no "it gets in when there is room". It also acts object by object, so nothing stops it admitting N-1 pods of the job and rejecting the last one. `LimitRange` only sets defaults and min/max per object. They are consumption guardrails; **they are not a scheduler**.214215`schedulingGates` (Pod Scheduling Readiness, **GA in 1.30**) and `Job.spec.suspend` are the *hooks* on which a queue manager is built — they hold individual pods or objects so an external integrator can decide. They provide no group semantics by themselves.216217**The two real options** (verified Aug 2026; both Apache-2.0 read in raw):218219| | **Kueue** | **Volcano** |220|---|---|---|221| What it is | **Job-level** queue manager on top of the default scheduler | **Full batch scheduler**, replacing/added to the default one |222| Version | **v0.19.0** (2026-07-22) | **v1.15.1** (2026-07-30) |223| Governance | `kubernetes-sigs`, **official SIG Scheduling subproject** | **CNCF Incubating** (accepted 2020-04-09, incubating 2022-03-21) |224| API | `kueue.x-k8s.io/**v1beta2**` (v1beta1 deprecated, non-storage). **There is no v1 GA** | `scheduling.volcano.sh/v1beta1` (PodGroup, Queue) and `batch.volcano.sh/**v1alpha1**` (Job) |225| CRDs | ClusterQueue, Cohort, AdmissionCheck, Topology (cluster) · LocalQueue, Workload (namespaced) | PodGroup (ns), Queue (cluster), Job (ns) |226| All-or-nothing | `waitForPodsReady` in the controller's Configuration | `gang` plugin over `PodGroup.minMember`/`minResources` |227| Requirement | K8s ≥ 1.29 | — |228229- **Kueue** suspends and queues the Workload until there is quota; `waitForPodsReady` (30 min timeout by default, `recoveryTimeout`, `requeuingStrategy` with backoff) evicts and requeues the job if not all its pods start. With `blockAdmission: true` it admits **sequentially** to break the deadlock — at the cost of serialising starts even when there is plenty of room. It is a mitigation, not strict gang scheduling.230- **Volcano** does real gang: `minMember` — *"if there's not enough resources to start all tasks, the scheduler will not start anyone"* — plus fair-share between `Queue`s and scheduling policies (binpack, NUMA, topology). The price: **a second scheduler in the cluster** and a Job API still on `v1alpha1` despite years in incubation. Weigh it up.231- **Criteria**: if what is missing is **queues, per-team quotas and job admission**, Kueue (you keep a single scheduler and it fits `Job`, JobSet and the training operators). If you need **strict gang and scheduling policies**, Volcano. **One per organisation, not both**: two schedulers deciding over the same node pool is an incident waiting to happen; if they coexist, partition by taints and be explicit about which nodes each one manages.232- **JobSet** (`jobset.x-k8s.io/v1alpha2`, v0.12.0, sig-apps) groups several `Job`s as a unit for HPC/ML. Still **alpha**: it complements Kueue, it does not replace it.233- **Forbidden to simulate gang by hand**: `initContainers` waiting for their peers, creative `podAffinity` or startup sleeps are the same deadlock with more steps and no observability.234235**When the right answer is not to use Kubernetes.** If the batch is classic HPC —tight MPI, low-latency interconnect, whole-node reservation, accounting by CPU-hours, users submitting with `sbatch`— **the answer is usually Slurm, not a Kubernetes cluster with two operators on top**. Those criteria belong to `hpc-standards` and are consulted there before building anything here. The practical boundary: if what you already have lives in Kubernetes and all that is missing are queues, Kueue/Volcano; if what you have is a computing centre, do not reimplement it in YAML.236237## 7. Sustainability and prohibitions238239- **Upgrade cadence**: K8s publishes 3 minors/year and supports N-2 → plan **at least 2 minor upgrades per year**; never more than one minor behind the managed provider's support. Before upgrading: read the deprecations (`kubectl api-resources`, Pluto for removed APIs), update charts/operators, rehearse in staging.240- Argo CD/Flux/Kyverno: stay within the 3 supported minors (quarterly review cadence).241- Refresh base images even if the app does not change (scheduled weekly/monthly rebuild: the CVEs arrive on their own).242243**FORBIDDEN** (automatic gate wherever possible):244- `:latest` or mutable tags in prod; images without a digest.245- Manual `kubectl apply`/`edit`/`scale`/`port-forward` against prod outside documented break-glass.246- Root containers, `privileged`, `hostNetwork`/`hostPID`/`hostPath` without a signed exception.247- Secrets in the clear in Git, in values, in the manifest's env or in logs.248- Workloads without requests/limits, without a readiness probe or without a PDB (multi-replica) in prod.249- Application namespaces without a default-deny NetworkPolicy or PSS `restricted`.250- An unsigned image or one without an SBOM on the path to prod.251- Ingress NGINX in new deployments (retired project, no security patches).252- Helm 2/3 in new projects; charts without `values.schema.json`.253- Tolerated drift: resources in the cluster that do not exist in Git.254- A mixed Linux/Windows cluster with Pods lacking `.spec.os.name`, lacking a `kubernetes.io/os` `nodeSelector` or with untainted Windows nodes; a Linux DaemonSet without a selector after adding the first Windows node.255- Taking the protection of a Linux `securityContext` (seccomp, capabilities, `readOnlyRootFilesystem`, `allowPrivilegeEscalation`) at face value on a Windows workload: it does not protect, it is ignored.256- A Windows image without a monthly rebuild aligned with the "B release"; a base image whose build does not match the node's.257- Gang scheduling simulated with `initContainers`, affinities or waits; two batch schedulers over the same node pool without partitioning.258259### Quick review checklist (every workload PR to prod)260261- [ ] Image by digest, signed, with an SBOM; distroless/minimal base and numeric non-root.262- [ ] requests/limits, readiness probe, PDB (if >1 replica), multi-AZ topology.263- [ ] Full restricted securityContext; namespace with PSS `restricted` + default-deny NetworkPolicy.264- [ ] No secrets in the clear; dedicated ServiceAccount with minimal RBAC and the token not mounted if unused.265- [ ] Render validated (kubeconform) + admission policies passed in CI; YAML diff visible in the PR.266- [ ] Metrics/logs/traces exposed; SIGTERM handled; rollback rehearsed or an expand/contract plan if there is a migration.267- [ ] If Windows: `.spec.os.name: windows`, `nodeSelector`+toleration, image build equal to the node's (`node.kubernetes.io/windows-build`), `runAsUserName` set, no Linux `securityContext` fields, and a monthly rebuild planned.268- [ ] If multi-pod batch: queue manager (Kueue or Volcano) declared and all-or-nothing semantics configured — not `ResourceQuota` as a substitute.269270## 8. Mandatory web verification271272Before pinning any version, API (`apiVersion`), flag or feature in deliverables:2731. **WebSearch/WebFetch** of the component's official release and endoflife.date (K8s, Helm, Argo CD/Flux, Kyverno, cosign) — the ecosystem rotates every quarter and this document ages.2742. Verify the K8s API deprecations of the target version jump (official release notes + Pluto).2753. Cosign: confirm whether v4 has shipped (it removes v3's deprecated flags) before writing signing pipelines.2764. **Windows**: re-verify on MS Learn (`version-compatibility`, `container-base-images`, `update-containers`) the host↔image matrix and the lifecycle before choosing a base — it is where the most stale information circulates. Check on `kubernetes.io/docs/concepts/windows/intro` which Windows Server versions are supported nodes on your K8s minor (as of 1.36: **only 2022 and 2025**) and whether the list of unsupported `securityContext` fields has changed. **WS2022 leaves mainstream on 2026-10-14**: from that date this table no longer describes a viable option for new nodes. The image sizes were measured from the MCR manifests in Aug 2026 and change with every "B release": measure them again (`/v2/<img>/manifests/<tag>` → sum of `layers[].size`), do not quote them from here.2775. **Batch**: re-verify the version and maturity of **Kueue** (is it still on `v1beta2` or is there a `v1` GA?) and of **Volcano** (is it still Incubating in the CNCF or has it graduated? is its `Job` still on `v1alpha1`?), and the status of **JobSet** (alpha as of Aug 2026). Cross-check the project's official site against `api.github.com`, not against the releases feed.2786. If you cannot verify, say so explicitly in the deliverable instead of assuming.279280**Declared gaps (Aug 2026), to be verified before using them as data**:281- The on-disk (uncompressed) sizes of the Windows images **were not measured**: the figures in the table are the registry's compressed sizes. On node disk they are significantly larger.282- Kueue's official installation page (`kueue.sigs.k8s.io/docs/installation/`) **could not be fetched**: v0.19.0 is backed by `api.github.com` + the Atom feed + the repo source at the tag, but **without confirmation from the project's website**.283- Of Kueue's CRDs, the manifests for ClusterQueue, AdmissionCheck, Topology and Workload were read; **`LocalQueue` and `ResourceFlavor` are assumed to follow the same `v1beta1` deprecated / `v1beta2` storage pattern by extrapolation**, not by direct reading.284- For the Volcano `Job`, the group, kind and `v1alpha1` version were confirmed, but **not the `served`/`storage` booleans** of the manifest. The chain `Job.minAvailable` → `PodGroup.minMember` is reasoned inference: the CRD does not document that field.285- The CNCF TAG that Volcano belongs to **was not verified** against a current primary source.286- The **v0.13.0** version of `windows-gmsa` comes from `api.github.com`; the project does not publish its own website to cross-check it against.287288If the web contradicts this document, **the web wins** — flag the discrepancy.
Run npx skillmds@latest add serialexperimentslainnnn/kubernetes-standards in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Kubernetes and container standards (staff/principal level). Use when writing or reviewing Dockerfiles/Containerfiles, OCI images, Kubernetes manifests (*.yaml with apiVersion/kind), Helm charts (Chart.yaml, values.yaml, templates/), Kustomize overlays (kustomization.yaml), GitOps configs (Argo CD Application, Flux Kustomization/HelmRelease), admission policies (Kyverno, OPA/Gatekeeper), image signing (cosign) or SBOM tooling. Also for Windows nodes and Windows containers (Server Core, Nano Server, ltsc2022/ltsc2025 base images, host/image version compatibility, runAsUserName, HostProcess, gMSA and GMSACredentialSpec) and for batch scheduling on Kubernetes (job queues, gang scheduling, Kueue, Volcano, ResourceQuota). It is listed under DevOps & Infra on SkillMD.
This skill has not completed SkillMD's automated safety review yet. Capability flags: reads secrets. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
serialexperimentslainnnn (@serialexperimentslainnnn) published this skill. Their other Agent Skills are listed on their SkillMD profile.