GKE Manifest Generation Skill
This skill provides guidelines, tooling integration, and templates to translate
natural language descriptions or application code changes into secure,
compliant, and cost-effective Kubernetes YAML manifests optimized for both GKE
Autopilot and GKE Standard clusters.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-manifest-generation)"
and append agent-skills/1.0 (skill:gke-manifest-generation) to User-Agent headers.
Core Rules & Verification
When generating or updating YAML manifests, you must strictly adhere to the
following rules:
1. Namespace & Resource Isolation
- Explicit Namespace: Always declare
namespace: {namespace} explicitly
in the metadata of every resource (Deployments, Services, ConfigMaps,
Secrets, PVCs, Roles, bindings). Map it to the namespace configured in your
active SETTINGS.md. Never omit the namespace.
- Dedicated ServiceAccount: Avoid using the namespace's
default
ServiceAccount. Always create and reference a dedicated ServiceAccount
(e.g., devteam-agent-sa) for each microservice.
2. GKE Resource Tuning (Autopilot & Standard)
Resources Requests & Limits: Always specify CPU and Memory requests and
limits for all containers.
- GKE Autopilot: Requests determine pod billing directly; requests and
limits must be equal. If they differ, Autopilot will automatically scale
requests up to match limits, which can significantly increase costs.
- GKE Standard: Requests ensure stable scheduling and bin-packing;
limits prevent resource starvation/noisy-neighbor issues.
Density Defaults: For stateless apps or sidecars on GKE Standard,
default to conservative requests (e.g., requests.cpu: "100m" or "200m",
requests.memory: "256Mi" or "512Mi") with burstable limits. Use a
reasonable overcommit ratio for limits (e.g., 2x to 4x requests, like
limits.cpu: "400m" to "800m", and limits.memory: "512Mi" to "1Gi").
Avoid excessive overcommit limits (like limits.cpu: "4" for a 100m
request) to prevent severe CPU throttling and latency degradation under
heavy scheduling load, particularly in environments without guaranteed node
shares.
Spot VMs for Staging/Dev: For non-production workloads (e.g., namespaces
containing -test, -dev, or -staging), or if the user requests cost
optimization, automatically target GKE Spot VMs. This requires injecting
both the nodeSelector targeting Spot VMs AND the corresponding toleration
to tolerate the Spot VM taint:
nodeSelector:
cloud.google.com/gke-spot: "true"
tolerations:
- key: "cloud.google.com/gke-spot"
operator: "Equal"
value: "true"
effect: "NoSchedule"
(On GKE Standard, this assumes a Spot node pool is configured).
3. Container Security Hardening (Pod Security Standards)
- Non-Root Execution: Always configure
securityContext at the Pod level
(and container level if overriding) to run as a non-root user (e.g.,
runAsNonRoot: true, runAsUser: 10000, runAsGroup: 10000, fsGroup: 10000). This is strictly enforced on GKE Autopilot and is a critical
security baseline for GKE Standard.
- Minimal Privileges: Always set
allowPrivilegeEscalation: false and
seccompProfile: {type: RuntimeDefault}.
- Read-Only Root Filesystem: Set
readOnlyRootFilesystem: true to prevent
modifications to the container image filesystem.
- Writable Directory Fallback: If
readOnlyRootFilesystem is enabled,
mount a local emptyDir volume to /tmp or /var/run/ to allow
applications (like Java/Nginx) to write temp files without crashing.
- Secret Volume Mounting: Prefer mounting Secrets as read-only files
(configured in the
volumes spec with defaultMode: 0400) instead of
mapping them as environment variables, unless the application framework
exclusively supports env-var based configuration. This prevents secrets
leaking into application logs.
4. Health Checking (Mandatory Probes)
Liveness & Readiness Probes: Every Deployment container must define both
livenessProbe and readinessProbe.
- Web/API: Use
httpGet probes.
- TCP Services: Use
tcpSocket probes.
- Databases/Caches: Use command-based
exec probes (e.g.,
exec.command: ["redis-cli", "ping"]).
Startup Probes for Slow-Starting Apps: For applications with slow boot
times (e.g., Java spring boot, complex Python scripts, LLM model servers),
you must also define a startupProbe. When a startupProbe is defined,
the liveness and readiness probes are disabled until it succeeds, preventing
Kubernetes from prematurely killing the pod during startup:
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
periodSeconds: 10
Sensible Defaults: Set initialDelaySeconds: 5 to 15 depending on
startup time (e.g., Java requires a longer delay than Go/Nginx).
5. Services & Ingress Routing
- Internal ClusterIP: Default all internal microservices to
type: ClusterIP. Never use type: LoadBalancer or NodePort unless the workload
is explicitly intended to be publicly accessible from the internet.
- Port Naming: Always assign clear, standard names to service and
container ports (e.g.,
name: http-web or name: grpc-api) to enable
automatic protocol discovery, tracing, and Web App routing.
- Prefer Gateway API: When exposing APIs externally, prioritize using GKE
Gateway API (
Gateway and HTTPRoute resources) over legacy Ingress
objects to enable advanced L7 routing and security features (e.g., Cloud
Armor).
6. Volume Mounts, StorageClasses & subPath Safety
- Avoid Directory Overwrites: When mounting a
ConfigMap or Secret to
an application directory containing other files (like Nginx public
directories), always use subPath to overlay only the specific file.
Caveat: Note that containers using subPath volume mounts do not receive
automatic configuration updates if the underlying ConfigMap or Secret is
modified; pods must be restarted manually to pick up changes.
- StorageClass Selection: Use the correct GKE storage class in
PersistentVolumeClaims:
- CSI Driver Clusters (Autopilot & Modern Standard): Use
standard-rwo
(default balanced PD) or premium-rwo (SSD PD).
- Legacy Standard Clusters: Use
standard (default PD) or premium
(SSD PD) if standard-rwo/premium-rwo are not configured.
- Database rule: Use SSD storage classes (
premium-rwo or premium)
only when the prompt explicitly requests high IOPS, low latency, or
database storage.
7. High Availability on GKE
- Topology Spread: For deployments with >1 replica, use
podAntiAffinity
or topologySpreadConstraints with topologyKey: "kubernetes.io/hostname"
to distribute pods across GKE nodes and availability zones.
- PodDisruptionBudget: For deployments with >1 replica, declare a
PodDisruptionBudget to guarantee minimum replica availability during
voluntary GKE node upgrades and maintenance cycles.
8. Updates & Server-Side Apply Reconciliations
- Stable List Keys: Under Kubernetes Server-Side Apply (SSA), elements in
associative lists (like volumes, volume mounts, ports, and container
definitions) are matched and merged by their unique identifier keys
(typically
name). You must keep the name key stable when modifying
properties of an existing list item. Renaming the name key will cause SSA
to create a brand new entry and leave the old entry intact (orphaned) rather
than modifying it.
- Minimal Diff: Make only the changes requested. Adhere closely to
existing labels, annotations, and conventions.
Specialty Workloads: GKE AI/Inference Serving (vLLM, TGI, etc.)
For model serving workloads, prioritize using optimized tooling like GKE
Inference Quickstart if available. If generating manually:
- GPU Request & Allocation:
- Always request
nvidia.com/gpu in both requests and limits.
- Add a
nodeSelector or node affinity targeting the desired GKE
accelerator tag (e.g., cloud.google.com/gke-accelerator: nvidia-l4).
- Shared Memory Boost:
- Model servers require high shared memory (
/dev/shm) for inter-process
communications. Always declare and mount an emptyDir volume with
medium: Memory to /dev/shm.
- Weight Loading Optimization:
- Mount model weight directories (like GCS buckets) using the GKE GCS Fuse
CSI driver (
csi.storage.gke.io) as readOnly: true for efficient
cold-starts.
Tooling & Grounding Guidelines
When generating manifests, you should leverage the following tooling to reduce
hallucinations and optimize configurations:
Inference Workloads (GKE Inference Quickstart CLI):
Make sure you have the
Google Cloud SDK installed.
For all AI/LLM inference workloads (e.g. model serving), you must
prioritize using the gcloud CLI GKE Inference Quickstart command to
generate the optimized manifests instead of writing them manually:
gcloud container ai profiles manifests create \
--model={model_name} \
--model-server={server_name} \
--accelerator-type={accelerator_type} \
--output=manifest \
--output-path={output_file_path}
Constraint: You must include all resources returned by this command
(Deployments, Services, PodMonitoring, etc.) without filtering.
Grounding in Official Documentation (Developer Knowledge API):
- For GKE-specific features, API defaults, manifest examples, or security
contexts, you must query Google's developer knowledge base to
retrieve official GKE documentation:
answer_query: Use this to ask direct questions (e.g., "How to
configure GCS Fuse CSI driver in GKE"). This is the preferred tool
for general queries.
search_documents: Use this to search for relevant GKE guides
or examples when you don't have a specific question.
get_document: Use this to fetch full document contents when
you have a specific document ID.
Reference Examples
For detailed, production-ready manifest templates, consult the following
reference guides:
- Basic Hardened Nginx Workload:
Production-ready deployment with dedicated service account, security
contexts, probes, anti-affinity, and PodDisruptionBudget.
- Network Policy: Default-deny ingress
network policy and selective ingress allowance for specific apps.
- AI/LLM Inference Workload: GPU resource
allocation, Workload Identity, GCS FUSE CSI driver mounting,
/dev/shm
shared memory boost, and startup probes.
- GKE Gateway API Routing: Exposing workloads
using GKE L7 Gateway API (
Gateway and HTTPRoute resources).
1---2description: Generates and updates secure, production-ready Kubernetes YAML manifests optimized for GKE Autopilot and GKE Standard clusters. Use when creating or modifying GKE deployment manifests, configuring container security contexts, setting CPU/memory resource limits, defining readiness/liveness/startup probes, mounting secrets and volumes, configuring GKE Gateway API routes, targeting Spot VMs, or deploying AI model inference workloads (vLLM, TGI, Gemma). Don't use for live cluster operations, pod troubleshooting (use gke-workload-troubleshooting), or cluster infrastructure provisioning (use gke-cluster-creation).3metadata:4 source: https://skillmd.com/skills/h3y6e/gke-manifest-generation5 category: Containers6 github-path: skills/cloud/gke-manifest-generation7 github-ref: refs/heads/main8 github-repo: https://github.com/google/skills9 github-tree-sha: 6633b9bdc6c9d78892000f553064690c681cde9210 version: 1.0.111name: gke-manifest-generation12---1314# GKE Manifest Generation Skill1516This skill provides guidelines, tooling integration, and templates to translate17natural language descriptions or application code changes into secure,18compliant, and cost-effective Kubernetes YAML manifests optimized for both GKE19Autopilot and GKE Standard clusters.2021## Attribution2223**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you24MUST read [Usage Attribution](references/attribution.md). For commands executed25directly in the terminal (never in generated scripts, Terraform, or user code),26prefix `gcloud` with27`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-manifest-generation)"`28and append `agent-skills/1.0 (skill:gke-manifest-generation)` to `User-Agent` headers.2930## Core Rules & Verification3132When generating or updating YAML manifests, you **must** strictly adhere to the33following rules:3435### 1. Namespace & Resource Isolation3637- **Explicit Namespace**: Always declare `namespace: {namespace}` explicitly38 in the metadata of every resource (Deployments, Services, ConfigMaps,39 Secrets, PVCs, Roles, bindings). Map it to the namespace configured in your40 active `SETTINGS.md`. Never omit the namespace.41- **Dedicated ServiceAccount**: Avoid using the namespace's `default`42 ServiceAccount. Always create and reference a dedicated `ServiceAccount`43 (e.g., `devteam-agent-sa`) for each microservice.4445### 2. GKE Resource Tuning (Autopilot & Standard)4647- **Resources Requests & Limits**: Always specify CPU and Memory requests and48 limits for all containers.49 - *GKE Autopilot*: Requests determine pod billing directly; requests and50 limits must be equal. If they differ, Autopilot will automatically scale51 requests up to match limits, which can significantly increase costs.52 - *GKE Standard*: Requests ensure stable scheduling and bin-packing;53 limits prevent resource starvation/noisy-neighbor issues.54- **Density Defaults**: For stateless apps or sidecars on GKE Standard,55 default to conservative requests (e.g., `requests.cpu: "100m"` or `"200m"`,56 `requests.memory: "256Mi"` or `"512Mi"`) with burstable limits. Use a57 reasonable overcommit ratio for limits (e.g., 2x to 4x requests, like58 `limits.cpu: "400m"` to `"800m"`, and `limits.memory: "512Mi"` to `"1Gi"`).59 Avoid excessive overcommit limits (like `limits.cpu: "4"` for a `100m`60 request) to prevent severe CPU throttling and latency degradation under61 heavy scheduling load, particularly in environments without guaranteed node62 shares.63- **Spot VMs for Staging/Dev**: For non-production workloads (e.g., namespaces64 containing `-test`, `-dev`, or `-staging`), or if the user requests cost65 optimization, automatically target GKE Spot VMs. This requires injecting66 both the `nodeSelector` targeting Spot VMs AND the corresponding toleration67 to tolerate the Spot VM taint:6869 ```yaml70 nodeSelector:71 cloud.google.com/gke-spot: "true"72 tolerations:73 - key: "cloud.google.com/gke-spot"74 operator: "Equal"75 value: "true"76 effect: "NoSchedule"77 ```7879 (On GKE Standard, this assumes a Spot node pool is configured).8081### 3. Container Security Hardening (Pod Security Standards)8283- **Non-Root Execution**: Always configure `securityContext` at the Pod level84 (and container level if overriding) to run as a non-root user (e.g.,85 `runAsNonRoot: true`, `runAsUser: 10000`, `runAsGroup: 10000`, `fsGroup:86 10000`). This is strictly enforced on GKE Autopilot and is a critical87 security baseline for GKE Standard.88- **Minimal Privileges**: Always set `allowPrivilegeEscalation: false` and89 `seccompProfile: {type: RuntimeDefault}`.90- **Read-Only Root Filesystem**: Set `readOnlyRootFilesystem: true` to prevent91 modifications to the container image filesystem.92 - *Writable Directory Fallback*: If `readOnlyRootFilesystem` is enabled,93 mount a local `emptyDir` volume to `/tmp` or `/var/run/` to allow94 applications (like Java/Nginx) to write temp files without crashing.95- **Secret Volume Mounting**: Prefer mounting Secrets as read-only files96 (configured in the `volumes` spec with `defaultMode: 0400`) instead of97 mapping them as environment variables, unless the application framework98 exclusively supports env-var based configuration. This prevents secrets99 leaking into application logs.100101### 4. Health Checking (Mandatory Probes)102103- **Liveness & Readiness Probes**: Every Deployment container must define both104 `livenessProbe` and `readinessProbe`.105 - **Web/API**: Use `httpGet` probes.106 - **TCP Services**: Use `tcpSocket` probes.107 - **Databases/Caches**: Use command-based `exec` probes (e.g.,108 `exec.command: ["redis-cli", "ping"]`).109- **Startup Probes for Slow-Starting Apps**: For applications with slow boot110 times (e.g., Java spring boot, complex Python scripts, LLM model servers),111 you **must** also define a `startupProbe`. When a `startupProbe` is defined,112 the liveness and readiness probes are disabled until it succeeds, preventing113 Kubernetes from prematurely killing the pod during startup:114115 ```yaml116 startupProbe:117 httpGet:118 path: /healthz119 port: 8080120 failureThreshold: 30121 periodSeconds: 10122 ```123124- **Sensible Defaults**: Set `initialDelaySeconds: 5` to `15` depending on125 startup time (e.g., Java requires a longer delay than Go/Nginx).126127### 5. Services & Ingress Routing128129- **Internal ClusterIP**: Default all internal microservices to `type:130 ClusterIP`. Never use `type: LoadBalancer` or `NodePort` unless the workload131 is explicitly intended to be publicly accessible from the internet.132- **Port Naming**: Always assign clear, standard names to service and133 container ports (e.g., `name: http-web` or `name: grpc-api`) to enable134 automatic protocol discovery, tracing, and Web App routing.135- **Prefer Gateway API**: When exposing APIs externally, prioritize using GKE136 Gateway API (`Gateway` and `HTTPRoute` resources) over legacy `Ingress`137 objects to enable advanced L7 routing and security features (e.g., Cloud138 Armor).139140### 6. Volume Mounts, StorageClasses & subPath Safety141142- **Avoid Directory Overwrites**: When mounting a `ConfigMap` or `Secret` to143 an application directory containing other files (like Nginx public144 directories), always use `subPath` to overlay only the specific file.145 *Caveat*: Note that containers using `subPath` volume mounts do not receive146 automatic configuration updates if the underlying ConfigMap or Secret is147 modified; pods must be restarted manually to pick up changes.148- **StorageClass Selection**: Use the correct GKE storage class in149 PersistentVolumeClaims:150 - *CSI Driver Clusters (Autopilot & Modern Standard)*: Use `standard-rwo`151 (default balanced PD) or `premium-rwo` (SSD PD).152 - *Legacy Standard Clusters*: Use `standard` (default PD) or `premium`153 (SSD PD) if `standard-rwo`/`premium-rwo` are not configured.154 - *Database rule*: Use SSD storage classes (`premium-rwo` or `premium`)155 only when the prompt explicitly requests high IOPS, low latency, or156 database storage.157158### 7. High Availability on GKE159160- **Topology Spread**: For deployments with >1 replica, use `podAntiAffinity`161 or `topologySpreadConstraints` with `topologyKey: "kubernetes.io/hostname"`162 to distribute pods across GKE nodes and availability zones.163- **PodDisruptionBudget**: For deployments with >1 replica, declare a164 `PodDisruptionBudget` to guarantee minimum replica availability during165 voluntary GKE node upgrades and maintenance cycles.166167### 8. Updates & Server-Side Apply Reconciliations168169- **Stable List Keys**: Under Kubernetes Server-Side Apply (SSA), elements in170 associative lists (like volumes, volume mounts, ports, and container171 definitions) are matched and merged by their unique identifier keys172 (typically `name`). You **must** keep the `name` key stable when modifying173 properties of an existing list item. Renaming the `name` key will cause SSA174 to create a brand new entry and leave the old entry intact (orphaned) rather175 than modifying it.176- **Minimal Diff**: Make only the changes requested. Adhere closely to177 existing labels, annotations, and conventions.178179--------------------------------------------------------------------------------180181## Specialty Workloads: GKE AI/Inference Serving (vLLM, TGI, etc.)182183For model serving workloads, prioritize using optimized tooling like GKE184Inference Quickstart if available. If generating manually:1851861. **GPU Request & Allocation**:187 - Always request `nvidia.com/gpu` in both `requests` and `limits`.188 - Add a `nodeSelector` or node affinity targeting the desired GKE189 accelerator tag (e.g., `cloud.google.com/gke-accelerator: nvidia-l4`).1902. **Shared Memory Boost**:191 - Model servers require high shared memory (`/dev/shm`) for inter-process192 communications. Always declare and mount an `emptyDir` volume with193 `medium: Memory` to `/dev/shm`.1943. **Weight Loading Optimization**:195 - Mount model weight directories (like GCS buckets) using the GKE GCS Fuse196 CSI driver (`csi.storage.gke.io`) as `readOnly: true` for efficient197 cold-starts.198199--------------------------------------------------------------------------------200201## Tooling & Grounding Guidelines202203When generating manifests, you should leverage the following tooling to reduce204hallucinations and optimize configurations:2052061. **Inference Workloads (GKE Inference Quickstart CLI)**:207208 - Make sure you have the209 [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) installed.210 - For all AI/LLM inference workloads (e.g. model serving), you **must**211 prioritize using the `gcloud` CLI GKE Inference Quickstart command to212 generate the optimized manifests instead of writing them manually:213214 ```bash215 gcloud container ai profiles manifests create \216 --model={model_name} \217 --model-server={server_name} \218 --accelerator-type={accelerator_type} \219 --output=manifest \220 --output-path={output_file_path}221 ```222223 - *Constraint*: You must include all resources returned by this command224 (Deployments, Services, PodMonitoring, etc.) without filtering.2252262. **Grounding in Official Documentation (Developer Knowledge API)**:227228 - For GKE-specific features, API defaults, manifest examples, or security229 contexts, you **must** query Google's developer knowledge base to230 retrieve official GKE documentation:231 - **`answer_query`**: Use this to ask direct questions (e.g., *"How to232 configure GCS Fuse CSI driver in GKE"*). This is the preferred tool233 for general queries.234 - **`search_documents`**: Use this to search for relevant GKE guides235 or examples when you don't have a specific question.236 - **`get_document`**: Use this to fetch full document contents when237 you have a specific document ID.238239--------------------------------------------------------------------------------240241## Reference Examples242243For detailed, production-ready manifest templates, consult the following244reference guides:245246- **[Basic Hardened Nginx Workload](references/basic-workload.md)**:247 Production-ready deployment with dedicated service account, security248 contexts, probes, anti-affinity, and PodDisruptionBudget.249- **[Network Policy](references/network-policy.md)**: Default-deny ingress250 network policy and selective ingress allowance for specific apps.251- **[AI/LLM Inference Workload](references/ai-inference.md)**: GPU resource252 allocation, Workload Identity, GCS FUSE CSI driver mounting, `/dev/shm`253 shared memory boost, and startup probes.254- **[GKE Gateway API Routing](references/gateway-api.md)**: Exposing workloads255 using GKE L7 Gateway API (`Gateway` and `HTTPRoute` resources).