Sota Kubernetes

SOTA Kubernetes Platform Security & Operations

Purpose

Engineer and audit the Kubernetes platform so that a compromised workload, a hostile chart, a leaked token, or a malicious controller cannot pivot to cluster-admin, read every Secret, or take the cluster down. This skill owns the layer above the pod: control plane and etcd, API server and kubelet hardening, RBAC and ServiceAccount identity, admission control and policy-as-code, GitOps controllers, operators/CRDs and admission webhooks, Helm/Kustomize at admission, multi-tenancy boundaries, cluster lifecycle, and K8s audit logging.

It does not re-teach pod isolation mechanics. Boundaries it defers:

• Pod securityContext, seccomp, AppArmor, capabilities, PSA pod-level fields → sota-sandboxing (rules/03 containers & microVMs; rules/01 boundaries). This skill owns the admission-time enforcement of those fields, not their internals.
• NetworkPolicy semantics, CNI choice, service mesh, mTLS → sota-network-security. This skill states only the requirement (default-deny per namespace, enforced at admission).
• OIDC/SSO and RBAC-role design methodology → sota-identity-access. This skill owns the K8s RBAC mechanics and escalation traps.
• Runtime/audit-log detection content (Falco/Tetragon rules, detections) → sota-detection-engineering. This skill owns producing and shipping the audit stream.
• Workload identity, Secret storage backends (ESO, sealed-secrets, CSI) → sota-secrets-management (rules/01, rules/02). This skill owns etcd encryption-at-rest.
• Cloud IAM, managed-K8s selection, DR → sota-cloud-infrastructure (rules/02, rules/04, rules/07).
• CI/CD provenance, image signing, IaC/GitOps pipeline, runtime ops → sota-devsecops (rules/02, rules/04, rules/06, rules/07).

Two modes. Pick one explicitly at the start of the task.

BUILD mode

Use when provisioning a cluster, writing RBAC/policies/manifests, configuring a GitOps controller, or installing an operator/chart.

1. Name the cluster topology first: managed (EKS/GKE/AKS) vs self-hosted (kubeadm, k3s/k0s, Talos), single- vs multi-tenant, who the threat actors are (workload → control plane? tenant → tenant? compromised CI → cluster?). Read the matching rules.
2. Least privilege by default: no wildcard RBAC verbs/resources, automountService AccountToken: false unless the pod calls the API, scoped AppProjects, operators get the narrowest RBAC that works. Loosen only with a comment stating why.
3. Encrypt etcd at rest with a KMS provider (rules/01) — Secrets are base64, not encrypted, by default. This is the single most common "we thought we were covered" gap.
4. Admission is fail-closed and ENFORCING, not auditing. A policy left in audit/ warn forever is documentation, not a control (rules/03). Ship the AUDIT→ENFORCE rollout plan with the policy.
5. GitOps is the only write path to the cluster. Humans propose via PR; the controller reconciles. Scope the controller's own privileges and AppProjects tightly (rules/04).
6. Plan the upgrade before you build: version skew, EOL date, CVE-response runbook, tested etcd restore (rules/01, rules/07).

Deliverables: topology + threat statement, the concrete manifests/configs, the AUDIT→ENFORCE rollout, and the residual-risk/assumptions list (cloud-managed control plane internals you cannot see, org IAM, CNI behavior).

AUDIT mode

Use when reviewing an existing cluster, its RBAC, policies, GitOps config, or operators.

Procedure: inventory the cluster surface (control plane flags or managed equivalent, RBAC graph, admission policies, GitOps controllers, operators/CRDs, namespaces/tenancy); for each, walk the relevant rules-file Audit checklist; verify empirically with kubectl, kubectl auth can-i, rbac-tool/krane, kube-bench, and helm template where possible; report findings in the format below. Do not report style nits as security findings.

Severity conventions

Severity is judged by reachability (anonymous > workload/tenant > contributor > admin) × yield (cluster-admin/all-Secrets > namespace compromise > info leak > availability).

Finding format

file:line | rule | severity | effort | fix

Where effort is one of trivial / small / medium / large, rule is the rules-file section (e.g. 02 §RBAC-wildcards), and fix is the concrete change. Expand below the line with: what is wrong, who exploits it and how (concrete attack path), and a snippet when short. Example:

clusterrole-ci.yaml:14 | 02 §escalate-verbs | Critical | small | remove the bind/escalate grant; scope to the 3 named ClusterRoles the controller actually applies
  Issue: ClusterRole bound to the CI ServiceAccount grants rbac.authorization.k8s.io/{bind,escalate} on clusterroles.
  Attack path: anyone who can run a job as this SA (any merged PR) can bind themselves cluster-admin.

End every audit with: counts per severity, top 3 fixes by risk-reduction-per-effort, and an explicit OUT-OF-SCOPE list (managed control-plane internals, org IAM, CNI/mesh, runtime detection content).

Rules index

When a task spans layers (most do), read every matching file. For a full cluster audit, read all seven plus the referenced sibling skills.

Top 10 non-negotiables

Violations are at minimum High in AUDIT mode and must never be introduced in BUILD mode:

1. anonymous-auth=false on the API server AND kubelet; kubelet read-only-port=0, kubelet authz mode Webhook (not AlwaysAllow). Anonymous or unauthenticated access to either is cluster-game-over (01).
2. etcd encrypted at rest with a KMS v2 provider (envelope DEK/KEK), not the default base64. Secrets are NOT encrypted out of the box (01, 06).
3. No wildcard RBAC. No * verbs/resources in Roles/ClusterRoles bound to workloads or users; no cluster-admin bound to a ServiceAccount, system:authenticated, or system:unauthenticated (02).
4. bind, escalate, impersonate are privilege-escalation verbs — grant only to a named, audited admin path, never to a workload or CI SA (02).
5. automountServiceAccountToken: false is the default; opt in only for pods that call the API. No long-lived Secret-based SA tokens — use bound/projected tokens (02).
6. Pod Security Admission restricted ENFORCED on workload namespaces (label pod-security.kubernetes.io/enforce: restricted), not merely warn/audit. PSA is stable; PSP is gone (03). Pod-level field internals → sota-sandboxing.
7. Admission policies enforce, with a documented AUDIT→ENFORCE rollout. A policy parked in Audit/warn indefinitely is a real, recurring finding (03).
8. GitOps is the only cluster write path; the controller and its AppProjects are tightly scoped. Never clusterResourceWhitelist: [{group:'*',kind:'*'}] with broad project access; pin Argo CD/Flux versions and patch known CVEs (04).
9. Operators get least-privilege RBAC; admission webhooks have correct failurePolicy/timeoutSeconds/namespaceSelector and TLS — a webhook is both a control and a single point of failure/attack (05).
10. API server audit logging enabled (RequestResponse for sensitive verbs/Secrets), shipped to tamper-resistant storage; etcd backups taken AND restore-tested (07,01). Detection content on the stream → sota-detection-engineering.

If the user asks for something that violates a non-negotiable, implement the secure alternative and explain the delta; comply only after they acknowledge the risk explicitly.