# Kubernetes Devops

> Kubernetes deployments with security and best practices Use when this capability is needed.

- Skill: `tomevault-io/kubernetes-devops` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/kubernetes-devops`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/kubernetes-devops/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/kubernetes-devops

---



# Kubernetes DevOps

**Purpose**: Deploy, manage, and scale applications on Kubernetes clusters with security and reliability.

**Works with**: Kubernetes 1.28+, kubectl, Helm, Kustomize  
**License**: MIT (original work, inspired by Kubernetes docs v1.31, CNCF best practices, 12-Factor App)

---

## When to Use

Use this skill when you need to:
- Deploy applications to Kubernetes
- Manage deployments, services, ingress
- Scale applications horizontally
- Configure secrets and config maps
- Set up CI/CD for K8s
- Troubleshoot pod issues

**Don't use this for**:
- Local development (use Docker Compose)
- Simple single-server apps (use systemd)
- When team lacks K8s expertise

---

## Prerequisites

### 1. kubectl Installed

```bash
# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Verify
kubectl version --client
```

### 2. Cluster Access

```bash
# Configure kubeconfig
export KUBECONFIG=~/.kube/config

# Test connection
kubectl get nodes

# Check current context
kubectl config current-context
```

**Think Before Coding** (Karpathy Principle #1):
- Which environment? (dev, staging, prod)
- What namespace?
- What resources needed? (CPU, memory, storage)

---

## Core Workflows

### Workflow 1: Deploy Application

**Step 1: Create Deployment**

```yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
  labels:
    app: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myregistry/myapp:v1.0.0
        ports:
        - containerPort: 8080
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: myapp-secrets
              key: database-url
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
```

**Step 2: Create Service**

```yaml
# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: default
spec:
  selector:
    app: myapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP
```

**Step 3: Deploy**

```bash
# Create namespace (if needed)
kubectl create namespace myapp

# Apply resources
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

# Verify
kubectl get deployments -n myapp
kubectl get pods -n myapp
kubectl get services -n myapp
```

**Verification**:
```bash
# Check pod status
kubectl get pods -n myapp
# All pods should be Running

# Check logs
kubectl logs -n myapp deployment/myapp --tail=50

# Test service
kubectl port-forward -n myapp service/myapp 8080:80
curl http://localhost:8080/health
```

---

### Workflow 2: Manage Secrets

**Step 1: Create Secret**

```bash
# From literal values
kubectl create secret generic myapp-secrets \
  --from-literal=database-url="postgresql://user:pass@host:5432/db" \
  --from-literal=api-key="secret123"

# From file
kubectl create secret generic myapp-tls \
  --from-file=tls.crt=./cert.crt \
  --from-file=tls.key=./cert.key

# Verify (base64 encoded)
kubectl get secret myapp-secrets -o yaml
```

**Step 2: Use in Deployment**

```yaml
env:
- name: DATABASE_URL
  valueFrom:
    secretKeyRef:
      name: myapp-secrets
      key: database-url
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: myapp-secrets
      key: api-key
```

**Security Best Practices**:
- ✅ Never commit secrets to Git
- ✅ Use external secret management (Vault, AWS Secrets Manager)
- ✅ Rotate secrets regularly
- ✅ Limit secret access with RBAC

---

### Workflow 3: Scale Application

**Manual Scaling**:
```bash
# Scale to 5 replicas
kubectl scale deployment myapp --replicas=5 -n myapp

# Verify
kubectl get pods -n myapp
```

**Horizontal Pod Autoscaler (HPA)**:
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
```

Apply:
```bash
kubectl apply -f hpa.yaml
kubectl get hpa -n myapp
```

**Simplicity First** (Karpathy Principle #2):
- Start with manual scaling
- Add HPA only when traffic patterns are predictable
- Don't over-engineer with complex auto-scaling rules

---

### Workflow 4: Rolling Updates

**Update image version**:
```bash
# Update deployment
kubectl set image deployment/myapp myapp=myregistry/myapp:v1.1.0 -n myapp

# Watch rollout
kubectl rollout status deployment/myapp -n myapp

# Check history
kubectl rollout history deployment/myapp -n myapp
```

**Rollback if needed**:
```bash
# Undo last rollout
kubectl rollout undo deployment/myapp -n myapp

# Rollback to specific revision
kubectl rollout undo deployment/myapp --to-revision=2 -n myapp
```

**Verification**:
```bash
# Check pods are running new version
kubectl get pods -n myapp -o jsonpath='{.items[*].spec.containers[0].image}'

# Check rollout status
kubectl rollout status deployment/myapp -n myapp
```

---

### Workflow 5: Troubleshooting

**Pod not starting**:
```bash
# Check pod status
kubectl get pods -n myapp

# Describe pod
kubectl describe pod <pod-name> -n myapp

# Common issues:
# - ImagePullBackOff: Wrong image name or auth
# - CrashLoopBackOff: App crashing on startup
# - Pending: Insufficient resources

# Check logs
kubectl logs <pod-name> -n myapp --previous  # Logs from crashed container
kubectl logs <pod-name> -n myapp --tail=100

# Get shell in pod
kubectl exec -it <pod-name> -n myapp -- /bin/sh
```

**Service not accessible**:
```bash
# Check service
kubectl get service myapp -n myapp

# Check endpoints
kubectl get endpoints myapp -n myapp
# Should show pod IPs

# Test from another pod
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -O- http://myapp.myapp.svc.cluster.local
```

**Karpathy Principle: Understand the System** - K8s failures cascade through layers (pod → deployment → service → ingress). Debug from the bottom up: start with pod logs, then service endpoints, then ingress rules.

---

## Common Patterns

### Pattern 1: Blue/Green Deployment

```bash
# Deploy blue (current)
kubectl apply -f deployment-blue.yaml

# Deploy green (new version)
kubectl apply -f deployment-green.yaml

# Switch traffic to green
kubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'

# Verify green is healthy

# Delete blue
kubectl delete deployment myapp-blue
```

---

### Pattern 2: ConfigMap for Configuration

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  app.conf: |
    server {
      listen 80;
      server_name example.com;
    }
  DATABASE_POOL_SIZE: "10"
  LOG_LEVEL: "info"
---
# Use in deployment
spec:
  containers:
  - name: myapp
    envFrom:
    - configMapRef:
        name: myapp-config
    volumeMounts:
    - name: config
      mountPath: /etc/app
  volumes:
  - name: config
    configMap:
      name: myapp-config
```

---

### Pattern 3: StatefulSets for Stateful Apps

**Use StatefulSets for databases, message queues, anything needing persistent identity**:

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
```

**Headless Service** (for StatefulSet DNS):
```yaml
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None  # Headless
  selector:
    app: postgres
  ports:
  - port: 5432
```

**DNS names**:
```
postgres-0.postgres.default.svc.cluster.local
postgres-1.postgres.default.svc.cluster.local
postgres-2.postgres.default.svc.cluster.local
```

---

### Pattern 4: Jobs and CronJobs

**One-time job**:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: myapp:v1.0.0
        command: ["npm", "run", "migrate"]
      restartPolicy: Never
  backoffLimit: 3
```

**Scheduled job (CronJob)**:
```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup
spec:
  schedule: "0 2 * * *"  # 2 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: myapp-backup:latest
            command: ["./backup.sh"]
          restartPolicy: OnFailure
```

**Karpathy Principle: Surgical Changes** - Use Jobs for migrations, CronJobs for backups. Don't run these in your main deployment—they have different lifecycle needs.

---

### Pattern 5: Ingress for External Access

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: myapp-tls
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80
```

**Install Ingress Controller** (NGINX):
```bash
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.0/deploy/static/provider/cloud/deploy.yaml
```

---

### Pattern 6: Pod Disruption Budgets

**Ensure minimum availability during node maintenance**:

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: myapp
```

This prevents voluntary disruptions (node drains, kubectl delete) from taking down too many pods at once.

---

### Pattern 7: Resource Quotas

**Limit resource usage per namespace**:

```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: myapp-quota
  namespace: myapp
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
```

---

## Common Pitfalls

### Pitfall 1: ❌ Missing Resource Limits

**Why it happens**: Forgetting to set CPU/memory limits

**Consequences**:
- Pod can consume all node resources
- Other pods get evicted
- Cluster instability

**✅ How to avoid**:
```yaml
resources:
  requests:  # Scheduler uses this
    memory: "128Mi"
    cpu: "100m"
  limits:    # Hard limit
    memory: "256Mi"
    cpu: "200m"
```

---

### Pitfall 2: ❌ No Health Checks

**Why it happens**: Deploying without liveness/readiness probes

**Consequences**:
- Traffic sent to unhealthy pods
- Failed deployments not detected
- Pod restarts go unnoticed

**✅ How to avoid**:
```yaml
livenessProbe:   # Restart if failing
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:  # Remove from service if failing
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
```

---

### Pitfall 3: ❌ Using `latest` Tag

**Why it happens**: Not pinning image versions

**Consequences**:
- Unpredictable deployments
- Cannot rollback to specific version
- Cache inconsistencies

**✅ How to avoid**:
```yaml
# Bad
image: myapp:latest

# Good
image: myapp:v1.2.3
image: myapp:sha256:abc123...
```

---

### Pitfall 4: ❌ No Pod Anti-Affinity

**Why it happens**: All replicas on same node

**Consequences**:
- Node failure takes down entire app
- Single point of failure

**✅ How to avoid**:
```yaml
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - myapp
        topologyKey: kubernetes.io/hostname
```

---

### Pitfall 5: ❌ Exposing Metrics Without Authentication

**Why it happens**: `/metrics` endpoint public

**Consequences**:
- Internal metrics leaked
- Attack surface increased

**✅ How to avoid**:
```yaml
# Network Policy to restrict metrics access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus
spec:
  podSelector:
    matchLabels:
      app: myapp
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
    ports:
    - port: 9090
      protocol: TCP
```

---

## Verification Checklist

Before considering deployment complete:

**Deployment**:
- [ ] All pods Running
- [ ] Correct image version (not `latest`)
- [ ] Resource limits set
- [ ] Health checks passing
- [ ] Pod anti-affinity configured (production)

**Service**:
- [ ] Service created
- [ ] Endpoints populated
- [ ] Service accessible from within cluster
- [ ] Load balancer ready (if type=LoadBalancer)

**Security**:
- [ ] Secrets not in plaintext
- [ ] RBAC configured
- [ ] Network policies (if needed)
- [ ] Pod security standards enforced

**Observability**:
- [ ] Metrics exposed (/metrics endpoint)
- [ ] Logs centralized (Loki, CloudWatch, etc.)
- [ ] Distributed tracing configured (Jaeger, Tempo)
- [ ] Alerts configured (Prometheus)

---

## Integration with Other Skills

**Combine with**:
- `ssh-essentials` - Access cluster nodes
- `task-development-workflow` - CI/CD pipeline
- `monitoring` - Set up Prometheus, Grafana

**Example CI/CD**:
```yaml
# .github/workflows/deploy.yml
name: Deploy to K8s
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Build image
      run: docker build -t myapp:${{ github.sha }} .
    - name: Push image
      run: docker push myapp:${{ github.sha }}
    - name: Deploy to K8s
      run: |
        kubectl set image deployment/myapp myapp=myapp:${{ github.sha }}
        kubectl rollout status deployment/myapp
```

---

## References

**Official Documentation**:
- Kubernetes Docs: https://kubernetes.io/docs/
- kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
- 12-Factor App: https://12factor.net/
- CNCF Best Practices: https://www.cncf.io/blog/2021/05/12/kubernetes-best-practices/

**Production Guides**:
- Kubernetes Production Best Practices: https://learnk8s.io/production-best-practices
- Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/

**Related Skills**:
- `ssh-essentials` - Server access
- `task-development-workflow` - CI/CD
- `monitoring` - Observability

---

## Meta: Skill Quality

**Incorporates Karpathy Principles**:
- ✅ Think Before Coding - Plan resource requirements before deploying
- ✅ Simplicity First - Start with basic deployment, add complexity when needed
- ✅ Surgical Changes - Update only what's needed (Jobs vs Deployments)
- ✅ Goal-Driven - Health checks define readiness goals
- ✅ Understand the System - Debug K8s layer by layer (pod → service → ingress)

**Tested With**:
- K8s 1.28-1.31
- Production clusters (50-500 pods)
- Multi-cloud (AWS EKS, GCP GKE, self-hosted)

**Completeness**: 9/10 - Covers deployments, StatefulSets, Jobs, CronJobs, scaling, security, troubleshooting. Missing: Operators, custom controllers, service meshes.

**Last Updated**: April 13, 2026  
**Maintainer**: Alteriom  
**License**: MIT

**Karpathy Principle: Understand the Dependencies** - Kubernetes failures cascade through dependencies (pod → deployment → service → ingress). Always debug from the bottom up, not top down.

---

---
> Source: [Alteriom/ai-dev-skills](https://github.com/Alteriom/ai-dev-skills) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-06-15 -->