Kubernetes API Patterns

Kubernetes API Patterns

Implement Kubernetes API patterns including Custom Resource Definition (CRD) development, webhook implementation, API groups, client library usage, and debugging techniques for custom API extensions.

TL;DR Checklist

• Define CRD API groups, versions, and resources following Kubernetes naming conventions
• Implement validation and conversion webhooks with proper admission review handling
• Choose correct client library (Go, Python, Java) based on deployment environment
• Use watch API with resource versions for efficient event streaming
• Debug API issues with kubectl explain, kubectl get --raw, and api-resources
• Migrate resources between API versions using conversion webhooks or sidecars
• Configure API server settings for custom resource scalability
• Implement subresources (status, scale) following Kubernetes patterns

When to Use

Use this skill when:

• Developing Custom Resource Definitions (CRDs) for domain-specific APIs
• Implementing admission webhooks (validating/mutating) for policy enforcement
• Creating client applications that interact with Kubernetes API resources
• Debugging API server issues or custom resource behavior
• Migrating resources between API versions during API evolution
• Extending Kubernetes with subresources (status, scale, scale status)
• Building controllers or operators that watch and reconcile custom resources

When NOT to Use

Avoid this skill for:

• Simple application deployment — Use Helm charts or kubectl apply instead of custom APIs
• Basic RBAC configuration — Use Kubernetes built-in roles instead of custom admission webhooks
• Logging or monitoring — Use dedicated operators like Prometheus Operator instead of custom APIs
• Serverless function triggers — Use Knative or eventing frameworks instead of CRDs
• Network policy management — Use CNI plugins with built-in policy support
• Large-scale batch processing — Use Tekton pipelines or Spark operators instead of custom CRDs

Core Workflow

1. Analyze API Requirements — Identify the domain objects, their relationships, and required operations. Checkpoint: Have you defined the API group, version, and resource names following Kubernetes conventions?

2. Design CRD Specification — Define the schema, validation rules, and subresources. Checkpoint: Does the schema prevent invalid states through type definitions?

3. Implement Webhooks (if needed) — Create admission webhooks for validation and mutation. Checkpoint: Do webhooks return clear admission review responses with proper error messages?

4. Choose Client Library — Select Go client for controllers, Python/Java for external tools. Checkpoint: Is the client version compatible with the Kubernetes cluster version?

5. Test API Interactions — Use kubectl and curl to verify API behavior. Checkpoint: Do all CRUD operations and watch functionality work as expected?

6. Debug and Optimize — Identify performance bottlenecks and fix issues. Checkpoint: Is the watch API using resource versions correctly to avoid missed events?

Implementation Patterns

Pattern 1: API Group Structure

Kubernetes organizes APIs into groups, versions, and resources. This structure enables versioning and deprecation.

BAD — Inconsistent naming and versioning:

# ❌ BAD — Mixed snake_case and camelCase, unclear versioning
apiVersion: myapp.io/v1beta1
kind: MyAppDeployment
spec:
  replicas: 3

GOOD — Proper group naming and versioning:

# ✅ GOOD — DNS-style group name, clear versioning
apiVersion: mycompany.com/v1
kind: MyAppDeployment
spec:
  replicas: 3

API Resource Structure:

API Group: mycompany.com
  Version: v1
    Resource: myappdeployments
      Scope: Namespaced

kubectl commands for API discovery:

# List all API groups
kubectl api-groups

# List resources in a specific group
kubectl api-resources --api-group=mycompany.com

# Show API version for a resource
kubectl explain myappdeployment --api-version=mycompany.com/v1

# Check if CRD is established
kubectl get crd myappdeployments.mycompany.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'

# List all resources across all groups
kubectl api-resources

# Get specific API version info
kubectl get --raw /apis/mycompany.com/v1

Pattern 2: CRD Definition with Validation

Custom Resource Definitions must include proper schema validation to prevent invalid states.

BAD — Missing schema validation:

# ❌ BAD — No validation allows invalid configurations
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myappdeployments.mycompany.com
spec:
  group: mycompany.com
  names:
    kind: MyAppDeployment
    plural: myappdeployments
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema: {}  # ❌ Empty schema allows any field

GOOD — Complete schema with validation:

# ✅ GOOD — Schema enforces valid configurations
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myappdeployments.mycompany.com
spec:
  group: mycompany.com
  names:
    kind: MyAppDeployment
    plural: myappdeployments
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required:
                - replicas
                - image
              properties:
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 100
                image:
                  type: string
                  pattern: "^[a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)*(:[a-z0-9]+([._-][a-z0-9]+)*)?(@[a-f0-9]{64})?$"
                resources:
                  type: object
                  properties:
                    limits:
                      type: object
                      properties:
                        memory:
                          type: string
                          pattern: "^[0-9]+(Mi|Gi|Ti)$"
                        cpu:
                          type: string
                          pattern: "^[0-9]+(m|)$"
                env:
                  type: array
                  items:
                    type: object
                    properties:
                      name:
                        type: string
                      value:
                        type: string

Create and verify CRD:

# Create CRD
kubectl apply -f myapp-crd.yaml

# Verify CRD is established
kubectl get crd myappdeployments.mycompany.com -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'

# List instances
kubectl get myappdeployments

# Describe CRD to see schema
kubectl describe crd myappdeployments.mycompany.com

# Validate CRD YAML before applying
kubectl apply --dry-run=client -f myapp-crd.yaml

Pattern 3: Validating Webhook Implementation

Admission webhooks intercept API requests to enforce custom policies.

BAD — Webhook with no error handling:

// ❌ BAD — No proper admission review response
func (h *ValidatingWebhook) Handle(ctx context.Context, req admission.Request) admission.Response {
    obj := &myappv1.MyAppDeployment{}
    if err := h.decoder.Decode(req.AdmissionReview.Spec.Object, obj); err != nil {
        return admission.Errored(http.StatusBadRequest, err)
    }
    
    // ❌ Missing proper validation response
    if obj.Spec.Replicas > 100 {
        // No admission response with denial
        return admission.Allowed("allowed")
    }
    
    return admission.Allowed("validation passed")
}

GOOD — Complete webhook with proper responses:

// ✅ GOOD — Proper admission review with clear deny messages
package webhook

import (
    "context"
    "fmt"
    "net/http"
    
    appsv1 "mycompany.com/api/v1"
    admissionv1 "k8s.io/api/admission/v1"
    admission "k8s.io/pod-security-admission/admission"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type ValidatingWebhook struct {
    decoder *admission.Decoder
}

func (h *ValidatingWebhook) Handle(ctx context.Context, req admission.Request) admission.Response {
    obj := &appsv1.MyAppDeployment{}
    if err := h.decoder.Decode(req.AdmissionReview.Spec.Object, obj); err != nil {
        return admission.Errored(http.StatusBadRequest, err)
    }

    // Validate replicas
    if obj.Spec.Replicas > 100 {
        return admission.Denied("replicas must be between 1 and 100")
    }

    // Validate image format
    if !isValidImage(obj.Spec.Image) {
        return admission.Denied(fmt.Sprintf("invalid image format: %s", obj.Spec.Image))
    }

    // Validate resource constraints
    if !h.hasResourceLimits(obj) {
        return admission.Denied("resource limits are required")
    }

    return admission.Allowed("validation passed")
}

func isValidImage(image string) bool {
    // Regex pattern for valid container image names
    pattern := `^[a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)*(:[a-z0-9]+([._-][a-z0-9]+)*)?(@[a-f0-9]{64})?$`
    // Implementation details...
    return true
}

func (h *ValidatingWebhook) hasResourceLimits(obj *appsv1.MyAppDeployment) bool {
    return obj.Spec.Resources.Limits != nil
}

// admission review response with uid and status
func admissionResponse(uid string, allowed bool, message string) admission.Response {
    return admission.Response{
        UID:     uid,
        Allowed: allowed,
        Result: &metav1.Status{
            Message: message,
        },
    }
}

Deploy webhook with service and certificate:

# Create TLS secret for webhook
openssl req -x509 -newkey rsa:4096 -keyout webhook-key.pem -out webhook-cert.pem -days 365 -nodes \
  -subj "/CN=myapp-validating-webhook.default.svc"

kubectl create secret generic webhook-tls --from-file=cert.pem=webhook-cert.pem --from-file=key.pem=webhook-key.pem

# Apply webhook configuration
kubectl apply -f validating-webhook.yaml

# Verify webhook is registered
kubectl get validatingwebhookconfigurations

# Test webhook manually
kubectl apply -f test-deployment.yaml --dry-run=client

# Check webhook logs
kubectl logs -n default deploy/myapp-validating-webhook

Pattern 4: Client Library Usage (Go)

The Go client is the standard for Kubernetes controllers and operators.

BAD — Inefficient client usage:

// ❌ BAD — Creates new client for each operation
func getDeployment(clientset *kubernetes.Clientset, name string) (*appsv1.Deployment, error) {
    // ❌ Creates new RESTClient each time
    config, _ := rest.InClusterConfig()
    client, _ := kubernetes.NewForConfig(config)
    
    return client.AppsV1().Deployments("default").Get(context.TODO(), name, metav1.GetOptions{})
}

GOOD — Reusing client with proper error handling:

// ✅ GOOD — Single client instance with proper configuration
package client

import (
    "context"
    "fmt"
    
    appsv1 "k8s.io/api/apps/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
)

var clientset *kubernetes.Clientset

// InitClient initializes the client once
func InitClient() error {
    config, err := rest.InClusterConfig()
    if err != nil {
        return fmt.Errorf("failed to get in-cluster config: %w", err)
    }
    
    clientset, err = kubernetes.NewForConfig(config)
    if err != nil {
        return fmt.Errorf("failed to create client: %w", err)
    }
    
    return nil
}

// GetDeployment retrieves a deployment with proper error handling
func GetDeployment(ctx context.Context, namespace, name string) (*appsv1.Deployment, error) {
    if clientset == nil {
        return nil, fmt.Errorf("client not initialized")
    }
    
    deployment, err := clientset.AppsV1().Deployments(namespace).Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("failed to get deployment %s/%s: %w", namespace, name, err)
    }
    
    return deployment, nil
}

// CreateDeployment creates a new deployment
func CreateDeployment(ctx context.Context, namespace string, deployment *appsv1.Deployment) (*appsv1.Deployment, error) {
    if clientset == nil {
        return nil, fmt.Errorf("client not initialized")
    }
    
    result, err := clientset.AppsV1().Deployments(namespace).Create(ctx, deployment, metav1.CreateOptions{})
    if err != nil {
        return nil, fmt.Errorf("failed to create deployment: %w", err)
    }
    
    return result, nil
}

// UpdateDeployment updates an existing deployment
func UpdateDeployment(ctx context.Context, namespace string, deployment *appsv1.Deployment) (*appsv1.Deployment, error) {
    if clientset == nil {
        return nil, fmt.Errorf("client not initialized")
    }
    
    result, err := clientset.AppsV1().Deployments(namespace).Update(ctx, deployment, metav1.UpdateOptions{})
    if err != nil {
        return nil, fmt.Errorf("failed to update deployment: %w", err)
    }
    
    return result, nil
}

// DeleteDeployment deletes a deployment
func DeleteDeployment(ctx context.Context, namespace, name string) error {
    if clientset == nil {
        return fmt.Errorf("client not initialized")
    }
    
    err := clientset.AppsV1().Deployments(namespace).Delete(ctx, name, metav1.DeleteOptions{})
    if err != nil {
        return fmt.Errorf("failed to delete deployment %s/%s: %w", namespace, name, err)
    }
    
    return nil
}

CRD client with dynamic client:

// ✅ GOOD — Dynamic client for unknown resource types
package client

import (
    "context"
    "fmt"
    
    apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
    "k8s.io/apimachinery/pkg/api/meta"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime/schema"
    "k8s.io/client-go/dynamic"
    "k8s.io/client-go/rest"
)

var dynamicClient dynamic.Interface

// InitDynamicClient initializes dynamic client for CRDs
func InitDynamicClient() error {
    config, err := rest.InClusterConfig()
    if err != nil {
        return fmt.Errorf("failed to get in-cluster config: %w", err)
    }
    
    dynamicClient, err = dynamic.NewForConfig(config)
    if err != nil {
        return fmt.Errorf("failed to create dynamic client: %w", err)
    }
    
    return nil
}

func getCustomResource(client dynamic.Interface, gvr schema.GroupVersionResource, namespace, name string) (*unstructured.Unstructured, error) {
    resource, err := client.Resource(gvr).Namespace(namespace).Get(context.TODO(), name, metav1.GetOptions{})
    if err != nil {
        return nil, fmt.Errorf("failed to get %s/%s: %w", gvr.Resource, name, err)
    }
    
    return resource, nil
}

// ListCustomResources lists all instances of a CRD
func ListCustomResources(gvr schema.GroupVersionResource, namespace string) (*unstructured.UnstructuredList, error) {
    if dynamicClient == nil {
        return nil, fmt.Errorf("dynamic client not initialized")
    }
    
    list, err := dynamicClient.Resource(gvr).Namespace(namespace).List(context.TODO(), metav1.ListOptions{})
    if err != nil {
        return nil, fmt.Errorf("failed to list %s: %w", gvr.Resource, err)
    }
    
    return list, nil
}

// UpdateCustomResourceStatus updates the status subresource
func UpdateCustomResourceStatus(gvr schema.GroupVersionResource, namespace, name string, status interface{}) (*unstructured.Unstructured, error) {
    if dynamicClient == nil {
        return nil, fmt.Errorf("dynamic client not initialized")
    }
    
    // Get current resource
    obj, err := dynamicClient.Resource(gvr).Namespace(namespace).Get(context.TODO(), name, metav1.GetOptions{})
    if err != nil {
        return nil, err
    }
    
    // Update status
    err = obj.Object["status"].(map[string]interface{})
    obj.Object["status"] = status
    
    return dynamicClient.Resource(gvr).Namespace(namespace).UpdateStatus(context.TODO(), obj, metav1.UpdateOptions{})
}

Pattern 5: Watch API with Resource Versions

The watch API efficiently streams resource changes.

BAD — Watch without resource version:

// ❌ BAD — Watch without resource version can miss events
func watchDeployments(client appsv1client.AppsV1Interface) {
    // ❌ No resourceVersion - can miss events during restart
    watcher, err := client.Deployments("default").Watch(context.TODO(), metav1.ListOptions{})
    if err != nil {
        panic(err)
    }
    
    for event := range watcher.ResultChan() {
        // Process event
        deployment := event.Object.(*appsv1.Deployment)
        fmt.Printf("Event: %s %s\n", event.Type, deployment.Name)
    }
}

GOOD — Watch with resource version tracking:

// ✅ GOOD — Watch with proper resource version handling
package watcher

import (
    "context"
    "fmt"
    "time"
    
    appsv1 "k8s.io/api/apps/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/watch"
)

// WatchDeployments watches deployments with resync capability
func WatchDeployments(client appsv1client.AppsV1Interface, resyncPeriod time.Duration) error {
    // Get current resource version
    list, err := client.Deployments("default").List(context.TODO(), metav1.ListOptions{Limit: 1})
    if err != nil {
        return fmt.Errorf("failed to list deployments: %w", err)
    }
    
    var resourceVersion string
    if len(list.Items) > 0 {
        resourceVersion = list.ListMeta.ResourceVersion
    }
    
    // Start watch with resource version
    watcher, err := client.Deployments("default").Watch(context.TODO(), metav1.ListOptions{
        ResourceVersion: resourceVersion,
        TimeoutSeconds:  ptr[int64](300), // 5 minute timeout
    })
    if err != nil {
        return fmt.Errorf("failed to start watch: %w", err)
    }
    
    // Handle events
    go func() {
        for event := range watcher.ResultChan() {
            deployment := event.Object.(*appsv1.Deployment)
            fmt.Printf("Event: %s %s (resourceVersion: %s)\n", 
                event.Type, deployment.Name, deployment.ResourceVersion)
        }
    }()
    
    // Periodic resync to catch missed events
    go func() {
        ticker := time.NewTicker(resyncPeriod)
        defer ticker.Stop()
        
        for range ticker.C {
            performResync(client)
        }
    }()
    
    return nil
}

// RestartWatch restarts watch with latest resource version
func restartWatch(client appsv1client.AppsV1Interface) error {
    list, err := client.Deployments("default").List(context.TODO(), metav1.ListOptions{Limit: 1})
    if err != nil {
        return err
    }
    
    var resourceVersion string
    if len(list.Items) > 0 {
        resourceVersion = list.ListMeta.ResourceVersion
    }
    
    watcher, err := client.Deployments("default").Watch(context.TODO(), metav1.ListOptions{
        ResourceVersion: resourceVersion,
        TimeoutSeconds:  ptr[int64](300),
    })
    if err != nil {
        return err
    }
    
    // Continue processing...
    return nil
}

func ptr[T any](v T) *T {
    return &v
}

func performResync(client appsv1client.AppsV1Interface) {
    deployments, err := client.Deployments("default").List(context.TODO(), metav1.ListOptions{})
    if err != nil {
        fmt.Printf("Resync failed: %v\n", err)
        return
    }
    
    for _, d := range deployments.Items {
        fmt.Printf("Resynced deployment: %s (replicas: %d)\n", d.Name, d.Spec.Replicas)
    }
}

Debug watch issues with kubectl:

# Watch events in a namespace
kubectl get events -n default --watch

# Get resource version for debugging
kubectl get deployments -o jsonpath='{.metadata.resourceVersion}'

# Watch with specific resource version
kubectl get deployments --watch --resource-version=<version>

# Check API server logs for watch errors
kubectl logs -n kube-system kube-apiserver-<node> | grep -i watch

# Test watch with curl
kubectl proxy &
curl -s --raw "http://localhost:8001/api/v1/namespaces/default/pods?watch=true&resourceVersion=0" &

Pattern 6: Subresource Implementation

Subresources extend resources with additional endpoints (status, scale).

BAD — Incomplete subresource configuration:

# ❌ BAD — Missing subresource specification
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myappdeployments.mycompany.com
spec:
  group: mycompany.com
  names:
    kind: MyAppDeployment
    plural: myappdeployments
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
            status:
              type: object
              # ❌ Status not defined as subresource

GOOD — Complete subresource configuration:

# ✅ GOOD — Status and scale subresources properly defined
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myappdeployments.mycompany.com
spec:
  group: mycompany.com
  names:
    kind: MyAppDeployment
    plural: myappdeployments
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 100
            status:
              type: object
              properties:
                replicas:
                  type: integer
                availableReplicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
                      reason:
                        type: string
                      message:
                        type: string
                      lastTransitionTime:
                        type: string
                        format: date-time
      subresources:
        status: {}  # ✅ Enables /status endpoint
        scale:
          specReplicasPath: .spec.replicas  # ✅ Enables /scale endpoint
          statusReplicasPath: .status.replicas
  additionalPrinterColumns:
    - name: Replicas
      type: integer
      description: Number of replicas
      jsonPath: .spec.replicas
    - name: Status
      type: string
      description: Current status
      jsonPath: .status.conditions[?(@.type=="Ready")].status
    - name: Age
      type: date
      jsonPath: .metadata.creationTimestamp

Subresource operations:

# Update status subresource
kubectl patch myappdeployment myapp --type=merge -p '
{
  "status": {
    "replicas": 3,
    "availableReplicas": 3,
    "conditions": [{"type": "Ready", "status": "True"}]
  }
}'

# Get scale subresource
kubectl get myappdeployment myapp -o jsonpath='{.spec.replicas}'

# Update scale subresource
kubectl patch myappdeployment myapp --type=merge -p '{"spec":{"replicas":5}}'

# Use kubectl scale command
kubectl scale myappdeployment myapp --replicas=5

# Get status subresource directly
kubectl get myappdeployment myapp -o json | jq '.status'

# Watch status changes
kubectl get myappdeployment myapp --watch --field-selector=metadata.name=myapp

Pattern 7: API Conversion Webhooks

Conversion webhooks enable smooth API version transitions.

BAD — No conversion webhook for version migration:

# ❌ BAD — Missing conversion configuration
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myappdeployments.mycompany.com
spec:
  group: mycompany.com
  names:
    kind: MyAppDeployment
    plural: myappdeployments
  scope: Namespaced
  versions:
    - name: v1beta1
      served: true
      storage: false
    - name: v1
      served: true
      storage: true
      # ❌ No conversion webhook - manual migration required

GOOD — Complete conversion webhook setup:

# ✅ GOOD — Conversion webhook for API version migration
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myappdeployments.mycompany.com
spec:
  group: mycompany.com
  names:
    kind: MyAppDeployment
    plural: myappdeployments
  scope: Namespaced
  versions:
    - name: v1beta1
      served: true
      storage: false
    - name: v1
      served: true
      storage: true
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: myapp-conversion-webhook
          namespace: default
          path: /convert
        caBundle: <base64-encoded-ca-cert>
      conversionReviewVersions:
        - v1
        - v1beta1

Conversion webhook implementation:

// ✅ GOOD — Webhook handles conversion between versions
package conversion

import (
    "context"
    "fmt"
    
    myappv1 "mycompany.com/api/v1"
    myappv1beta1 "mycompany.com/api/v1beta1"
    apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
    admission "k8s.io/pod-security-admission/admission"
)

type ConversionWebhook struct {
    decoder *admission.Decoder
}

func (h *ConversionWebhook) Handle(ctx context.Context, req admission.Request) admission.Response {
    review := &apiextensionsv1.ConversionReview{}
    if err := h.decoder.Decode(req.AdmissionReview.Spec.Object, review); err != nil {
        return admission.Errored(http.StatusBadRequest, err)
    }

    response := &apiextensionsv1.ConversionResponse{
        Uid: review.Request.UID,
    }

    switch review.Request.DesiredAPIVersion {
    case "mycompany.com/v1":
        response.ConvertedObjects, response.Status = h.convertToV1(review.Request.Objects)
    case "mycompany.com/v1beta1":
        response.ConvertedObjects, response.Status = h.convertToV1Beta1(review.Request.Objects)
    default:
        response.Status = &metav1.Status{
            Status:  metav1.StatusFailure,
            Code:    http.StatusBadRequest,
            Message: fmt.Sprintf("unsupported API version: %s", review.Request.DesiredAPIVersion),
        }
        return admission.Response{UID: review.Request.UID, Allowed: false, Result: response.Status}
    }

    return admission.Response{UID: review.Request.UID, Allowed: true, Result: response}
}

func (h *ConversionWebhook) convertToV1(objects []runtime.RawExtension) ([]runtime.RawExtension, *metav1.Status) {
    var converted []runtime.RawExtension
    
    for _, obj := range objects {
        v1beta1Obj := &myappv1beta1.MyAppDeployment{}
        if err := json.Unmarshal(obj.Raw, v1beta1Obj); err != nil {
            return nil, &metav1.Status{
                Status:  metav1.StatusFailure,
                Code:    http.StatusBadRequest,
                Message: fmt.Sprintf("failed to unmarshal v1beta1 object: %v", err),
            }
        }
        
        v1Obj := convertV1Beta1ToV1(v1beta1Obj)
        data, err := json.Marshal(v1Obj)
        if err != nil {
            return nil, &metav1.Status{
                Status:  metav1.StatusFailure,
                Code:    http.StatusInternalServerError,
                Message: fmt.Sprintf("failed to marshal v1 object: %v", err),
            }
        }
        
        converted = append(converted, runtime.RawExtension{Raw: data})
    }
    
    return converted, nil
}

func (h *ConversionWebhook) convertToV1Beta1(objects []runtime.RawExtension) ([]runtime.RawExtension, *metav1.Status) {
    var converted []runtime.RawExtension
    
    for _, obj := range objects {
        v1Obj := &myappv1.MyAppDeployment{}
        if err := json.Unmarshal(obj.Raw, v1Obj); err != nil {
            return nil, &metav1.Status{
                Status:  metav1.StatusFailure,
                Code:    http.StatusBadRequest,
                Message: fmt.Sprintf("failed to unmarshal v1 object: %v", err),
            }
        }
        
        v1beta1Obj := convertV1ToV1Beta1(v1Obj)
        data, err := json.Marshal(v1beta1Obj)
        if err != nil {
            return nil, &metav1.Status{
                Status:  metav1.StatusFailure,
                Code:    http.StatusInternalServerError,
                Message: fmt.Sprintf("failed to marshal v1beta1 object: %v", err),
            }
        }
        
        converted = append(converted, runtime.RawExtension{Raw: data})
    }
    
    return converted, nil
}

func convertV1Beta1ToV1(v1beta1Obj *myappv1beta1.MyAppDeployment) *myappv1.MyAppDeployment {
    return &myappv1.MyAppDeployment{
        TypeMeta: v1beta1Obj.TypeMeta,
        ObjectMeta: v1beta1Obj.ObjectMeta,
        Spec: myappv1.MyAppDeploymentSpec{
            Replicas:  v1beta1Obj.Spec.Replicas,
            Image:     v1beta1Obj.Spec.Image,
            Resources: convertResources(v1beta1Obj.Spec.Resources),
        },
        Status: myappv1.MyAppDeploymentStatus{
            Replicas:            v1beta1Obj.Status.Replicas,
            AvailableReplicas:   v1beta1Obj.Status.AvailableReplicas,
            Conditions:          convertConditions(v1beta1Obj.Status.Conditions),
        },
    }
}

func convertV1ToV1Beta1(v1Obj *myappv1.MyAppDeployment) *myappv1beta1.MyAppDeployment {
    return &myappv1beta1.MyAppDeployment{
        TypeMeta: v1Obj.TypeMeta,
        ObjectMeta: v1Obj.ObjectMeta,
        Spec: myappv1beta1.MyAppDeploymentSpec{
            Replicas:  v1Obj.Spec.Replicas,
            Image:     v1Obj.Spec.Image,
            Resources: convertResourcesBack(v1Obj.Spec.Resources),
        },
        Status: myappv1beta1.MyAppDeploymentStatus{
            Replicas:            v1Obj.Status.Replicas,
            AvailableReplicas:   v1Obj.Status.AvailableReplicas,
            Conditions:          convertConditionsBack(v1Obj.Status.Conditions),
        },
    }
}

Test conversion:

# Get resource in v1beta1
kubectl get myappdeployment myapp -o json --api-version=mycompany.com/v1beta1

# Get same resource in v1 (triggers conversion webhook)
kubectl get myappdeployment myapp -o json --api-version=mycompany.com/v1

# Verify conversion in etcd
kubectl get myappdeployment myapp -o yaml

# Test conversion with curl
kubectl proxy &
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"apiVersion":"apiextensions.k8s.io/v1","kind":"ConversionReview","request":{"uid":"test","desiredAPIVersion":"mycompany.com/v1","objects":[]}}' \
  http://localhost:8001/apis/mycompany.com/v1/convert

Pattern 8: API Server Configuration

Proper API server configuration ensures custom resources work efficiently.

BAD — Default API server settings for high-volume CRs:

# ❌ BAD — No API server tuning for custom resources
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-apiserver
  namespace: kube-system
data:
  kube-apiserver.conf: |
    # ❌ Using default settings
    kube-apiserver \
      --etcd-servers=http://127.0.0.1:2379 \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger

GOOD — Optimized API server configuration:

# ✅ GOOD — Tuned for custom resource performance
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-apiserver
  namespace: kube-system
data:
  kube-apiserver.conf: |
    kube-apiserver \
      --etcd-servers=https://127.0.0.1:2379 \
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ValidatingAdmissionWebhook,MutatingAdmissionWebhook \
      --enable-bootstrap-token-auth=true \
      --max-requests-inflight=2000 \
      --max-requests-inflight=1500 \
      --max-requests-inflight=1000 \
      --max-requests-inflight=500 \
      --request-timeout=300s \
      --allow-privileged=true \
      --authorization-mode=Node,RBAC \
      --client-ca-file=/etc/kubernetes/pki/ca.crt \
      --enable-swagger-ui=true \
      --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
      --runtime-config=api/all=true \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

# Optimized etcd configuration for CRDs
  etcd.conf: |
    # Optimize etcd for CRD storage
    --auto-compaction-mode=periodic
    --auto-compaction-retention=1h
    --quota-backend-bytes=8589934592  # 8GB
    --max-request-bytes=15728640  # 15MB
    --listen-client-urls=https://0.0.0.0:2379
    --listen-peer-urls=https://0.0.0.0:2380

CRD controller configuration:

// ✅ GOOD — Controller with proper rate limiting and resync
package controller

import (
    "context"
    "time"
    
    "github.com/go-logr/logr"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
    "k8s.io/apimachinery/pkg/runtime/schema"
    "k8s.io/apimachinery/pkg/util/runtime"
    "k8s.io/apimachinery/pkg/util/wait"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/util/workqueue"
)

type Controller struct {
    client    dynamic.Interface
    informer  cache.SharedIndexInformer
    queue     workqueue.RateLimitingInterface
    logger    logr.Logger
    gvr       schema.GroupVersionResource
}

func NewController(client dynamic.Interface, informer cache.SharedIndexInformer, gvr schema.GroupVersionResource, logger logr.Logger) *Controller {
    return &Controller{
        client:  client,
        informer: informer,
        queue:   workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()),
        logger:  logger,
        gvr:     gvr,
    }
}

func (c *Controller) Start(stopCh <-chan struct{}) error {
    // Set up event handlers
    c.informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    c.onAdd,
        UpdateFunc: c.onUpdate,
        DeleteFunc: c.onDelete,
    })
    
    // Start informer
    go c.informer.Run(stopCh)
    
    // Process work items
    c.processWorkItems(stopCh)
    
    return nil
}

func (c *Controller) processWorkItems(stopCh <-chan struct{}) {
    for {
        select {
        case <-stopCh:
            return
        default:
            obj, shutdown := c.queue.Get()
            if shutdown {
                return
            }
            
            if err := c.reconcile(obj); err != nil {
                c.logger.Error(err, "reconciliation failed")
                if c.queue.NumRequeues(obj) < 5 {
                    c.queue.AddRateLimited(obj)
                } else {
                    c.queue.Forget(obj)
                    runtime.HandleError(err)
                }
            } else {
                c.queue.Forget(obj)
            }
        }
    }
}

func (c *Controller) reconcile(obj interface{}) error {
    defer c.queue.Done(obj)
    
    key, ok := obj.(string)
    if !ok {
        return nil
    }
    
    namespace, name, err := cache.SplitMetaNamespaceKey(key)
    if err != nil {
        return err
    }
    
    gvr := c.gvr
    resource, err := c.client.Resource(gvr).Namespace(namespace).Get(context.TODO(), name, metav1.GetOptions{})
    if err != nil {
        if apierrors.IsNotFound(err) {
            return nil // Object deleted
        }
        return err
    }
    
    // Reconciliation logic here
    return nil
}

func (c *Controller) onAdd(obj interface{}) {
    key, err := cache.MetaNamespaceKeyFunc(obj)
    if err != nil {
        runtime.HandleError(err)
        return
    }
    c.queue.Add(key)
}

func (c *Controller) onUpdate(old, new interface{}) {
    key, err := cache.MetaNamespaceKeyFunc(new)
    if err != nil {
        runtime.HandleError(err)
        return
    }
    c.queue.Add(key)
}

func (c *Controller) onDelete(obj interface{}) {
    key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj)
    if err != nil {
        runtime.HandleError(err)
        return
    }
    c.queue.Add(key)
}

Constraints

MUST DO

• Use DNS-style naming for API groups (e.g., mycompany.com, k8s.io)
• Define strict schema validation to prevent invalid states (Parse, Don't Validate)
• Implement admission webhooks with clear, actionable error messages
• Use resource versions in watch API to avoid missing events
• Include status subresource for controller-managed fields
• Handle conversion webhooks gracefully during API version migrations
• Set appropriate API server request limits based on CRD volume
• Test CRD upgrades with existing resources before production deployment
• Validate webhook TLS certificates before deployment
• Implement proper RBAC for webhook service accounts
• Use admission webhooks for validation, not business logic

MUST NOT DO

• Use reserved API groups like k8s.io, kubernetes.io for custom resources
• Skip schema validation even for "internal" CRDs
• Return generic error messages from admission webhooks
• Watch without resource version tracking in production
• Modify CRD schema in ways that break existing resources
• Deploy webhooks without TLS certificates
• Set replicas too high in CRD without proper API server tuning
• Use v1beta1 as storage version (always use stable v1)
• Skip webhook timeout handling (default 10s, max 30s)
• Block webhook operations for more than timeout duration

Output Template

When implementing Kubernetes API patterns, include:

1. CRD Specification — Complete CRD definition with schema, validation, and subresources
2. Webhook Configuration — Validating/mutating webhook setup with service and certificate
3. Client Code — Go client or other language client for resource operations
4. Deployment Scripts — Helm charts or YAML manifests for webhook deployment
5. Migration Plan — Steps for API version conversion and upgrade procedures
6. Debugging Commands — kubectl commands for troubleshooting API issues
7. Monitoring Configuration — Prometheus metrics for API server and webhook performance

Related Skills

References

• Kubernetes API Documentation
• Custom Resource Definitions
• Admission Webhooks
• API Versioning
• Client Libraries
• Watching Resources

Live References

These links are resolved at load time — content is fetched and inlined to provide the latest documentation.

• [Kubern

…(truncated)