# Kubernetes Operator

> Kubernetes cluster operator for the Marshian Galaxy home lab. Use when managing workloads, namespaces, node operations, Cilium networking, or kubectl/helm workflows on the k8s0–k8s4 cluster. Use when this capability is needed.

- Skill: `tomevault-io/kubernetes-operator` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/kubernetes-operator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/kubernetes-operator/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/kubernetes-operator

---


# Kubernetes Operator

## Overview
This skill manages the Marshian Galaxy Kubernetes cluster — a multi-node home lab running on Alpine Linux VMs with Cilium for networking. It covers day-to-day kubectl operations, workload management, node health, and the custom Nushell helpers in `nushell/bin/kubernetes.nu`.

## Cluster Topology

| Node | Role | OS |
| :--- | :--- | :--- |
| `k8s0`, `k8s1`, `k8s2` | Control Plane + Worker | Alpine Linux |
| `k8s3`, `k8s4` | Worker | Alpine Linux |
| `srv2` | NFS / Minecraft services VM | Alpine Linux |
| `wireguard` | VPN gateway VM | Alpine Linux |
| `pihole` | DNS / ad-blocker VM | Debian |

Service VIP accessibility is verified via `https://git.marsh.gg` (Cilium load balancer).

## Core Capabilities

### 1. Node Operations
- Check cluster health: `kubectl get nodes -o wide`
- Cordon before maintenance: `kubectl cordon <node>`
- Drain safely: `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force`
- Uncordon after: `kubectl uncordon <node>`
- For reboots of the full cluster, delegate to the `galaxy-rebooter` skill.

### 2. Workload Management
- Deploy/update: `kubectl apply -f <manifest>` or `helm upgrade --install`
- Rollout status: `kubectl rollout status deployment/<name> -n <ns>`
- Rollback: `kubectl rollout undo deployment/<name>`
- Logs: `kubectl logs -n <ns> <pod> --tail=100 -f`

### 3. Nushell Kubernetes Helpers
The file `nushell/bin/kubernetes.nu` contains custom commands. When editing:
- Follow Nushell typing conventions (see `nushell-orchestrator` skill).
- Keep outputs as structured tables so they pipe naturally into `where`, `sort-by`, etc.
- Avoid shelling out to `jq`; use Nushell's `from json` instead.

### 4. Cilium Networking
- Cilium is the CNI — use `cilium status` and `cilium connectivity test` to diagnose network issues.
- The cluster VIP for external services resolves via Cilium's LB; verify with `curl -s https://git.marsh.gg`.
- When adding new `LoadBalancer` services, ensure `externalTrafficPolicy` and IP pool annotations match the existing pattern.

### 5. Control Plane API Transience
During control plane node reboots, the Kubernetes API becomes temporarily unavailable. Any script that polls `kubectl get node` must:
- Wrap in `try { ... } catch { ... }` (Nushell) or equivalent retry logic.
- Wait for SSH to return before attempting kubectl commands against that node.
- Expect transient auth/refused/forbidden errors — treat them as retry triggers, not hard failures.

### 6. Common Troubleshooting
- Pod stuck `Pending`: check `kubectl describe pod <name>` for resource/affinity/taint issues.
- Node `NotReady`: check `kubectl describe node <name>` and SSH in to check kubelet logs (`rc-service kubelet status`).
- DNS failures: check pihole is up; `kubectl exec -it <pod> -- nslookup kubernetes.default`.

## Examples
- "Show me all pods across all namespaces that aren't running."
- "Add a new namespace for my monitoring stack and deploy a Helm chart into it."
- "One of the workers is NotReady — help me diagnose."
- "Update the kubernetes.nu helper to show resource requests alongside pod status."

---
> Source: [icub3d/dotfiles](https://github.com/icub3d/dotfiles) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-06-15 -->