tomevault-io Anton cluster health
Read-only ordered triage for the Kubernetes layer and above. Never mutates cluster state — no restarts, no force reconciles, no applies. If the triage surfaces a fix, hand off to the matching task skill (debug-flux-reconciliation, upgrade-talos-or-k8s, etc.).
Scope — what this skill owns
| Layer | Owner | Notes |
|---|---|---|
| 1. Hardware, OS, disks, network | talos-inspect |
Always start there if any sign the problem is below Flux |
| 2. Control plane, etcd quorum | talos-inspect |
Talos manages these as static pods |
| 3. CNI & cluster DNS | this skill | layer-3-cni-dns |
| 4. Flux (controllers, sources, root ks) | this skill | layer-4-flux |
| 5. Platform components | this skill | layer-5-platform |
| 6. App-layer summary | this skill | layer-6-apps |
Stop at the first red layer and drill in. Anything above a broken layer is noise.
Gate — rule out the OS layer first
Before running anything in this skill, ask: have nodes been flapping, did a node just reboot, is anyone reporting unreachability? If yes, run talos-inspect first. Do not try to diagnose Flux when the real problem is a bonded interface down on k8s-2.
Green-path one-liner
If you have 10 seconds and want a single pulse:
flux get ks -A --status-selector ready=false && \
flux get hr -A --status-selector ready=false
Two empty results means Flux believes everything is ok. Necessary but not sufficient — the silent killers do not show up in that output. Check those next.
Full triage flow (layer 3 → 6)
Layer 3 — CNI & DNS
Cilium agent must be Ready on every node; CoreDNS needs at least one Ready pod. A DNS failure masquerades as almost any app problem.
kubectl -n kube-system get ds cilium
kubectl -n kube-system get deploy coredns
kubectl -n kube-system get pods -l k8s-app=kube-dns
Details, DNS smoke test, common failures → layer-3-cni-dns.
Layer 4 — Flux
Controllers themselves, then root flux-system Kustomization, then sources.
flux check
flux -n flux-system get ks flux-system
flux get sources all -A
SOPS decryption errors surface as Kustomization ready=False with a decryption error message — see silent killers. Full flow → layer-4-flux.
Layer 5 — Platform (silent-killer territory)
cert-manager, External-Secrets + ClusterSecretStore onepassword-connect, envoy-gateway + Gateway Programmed condition, cloudflared tunnel, k8s-gateway.
kubectl get clustersecretstore onepassword-connect -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
kubectl get gateway -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,PROGRAMMED:.status.conditions[?(@.type=="Programmed")].status'
Details per component → layer-5-platform.
Layer 6 — Apps (only meaningful if 3-5 are green)
flux get ks -A --status-selector ready=false
flux get hr -A --status-selector ready=false
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | tail -20
Count failures; do not deep-dive here. Hand off individual failures to debug-flux-reconciliation or direct log reading. Details → layer-6-apps.
Silent killers — the four that don't show up as "unhealthy"
These all pass flux get ks/hr -A but still break things. Always check explicitly. Full probes with copy-paste commands → silent-killers.
- SOPS decryption error — a Kustomization is
ready=Falsewith adecryption error; usually means.sops.yaml+ age key mismatch after a rotation - ClusterSecretStore
NotReady— ESO can't talk to 1Password, everyExternalSecretis frozen silently - Gateway
Programmed=False— envoy-gateway Accepted the Gateway but failed to program it, HTTPRoutes attach but no traffic flows - Cloudflared tunnel disconnected — tunnel pod is Running but not registered; you just fixed a QUIC→HTTP/2 regression in commit
42822fc3, watch for this class again
What this skill does NOT do
- Does not mutate: no restarts, no
flux reconcile, nokubectl delete, notalosctl apply - Does not cover storage — Rook-Ceph was removed in commit
889a9662, the cluster has no stateful storage layer right now - Does not cover node/OS/etcd/disk/network — that's
talos-inspect - Does not deep-dive individual failing apps — hand off to
debug-flux-reconciliation - Does not report capacity or metric trends — only liveness/readiness
Remote access
Every kubectl/flux command assumes the kubeconfig generated by task bootstrap or task talos:generate-config. For off-LAN access specifics → anton-remote-access.
Source: wcygan/anton — distributed by TomeVault.