Goodsmileduck Kubernetes Operators — CRD and reconcile review
For pod-level debugging (CrashLoopBackOff, ImagePullBackOff, scheduling failures) see the kubernetes-operations skill. For Argo-managed deployments see argocd-operations.
Most "operator bugs" are reconcile-loop bugs, not Kubernetes bugs: missing finalizers, blocking calls, no requeue on transient error, status drift, RBAC over-grant. The three Python scripts shipped here catch the deterministic subset before code reaches a cluster.
When to invoke
Run the analyzers first — they're stdlib Python, fast, and surface most routine issues:
SKILL=plugins/kubernetes-skills/skills/kubernetes-operators
python3 "$SKILL/scripts/crd_validator.py" --crd config/crd/
python3 "$SKILL/scripts/reconcile_lint.py" --controller controllers/
python3 "$SKILL/scripts/operator_capability_audit.py" --operator-dir .
All three accept --format json. Triage by severity: FAIL blocks merge, WARN files an issue.
Pre-flight: is an operator the right shape?
Operators are for stateful, lifecycle-managed workloads. Reach for one when:
- The thing being managed has an external API (RDS, Kafka topics, GitHub repos).
- Day-2 operations are non-trivial (backup, restore, version upgrade, failover).
- A Helm chart + bash isn't enough — you need a controller that observes and re-acts.
Don't reach for an operator when:
| Want | Better tool |
|---|---|
| Run a workload | Deployment / StatefulSet / Job |
| Package + parameterize manifests | Helm chart or Kustomize |
| Sync repo → cluster | ArgoCD / Flux |
| External-resource lifecycle, no day-2 complexity | Terraform / Crossplane composition |
Core principle: reconcile is declarative, not imperative
The wrong shape:
if creating: do A
elif updating: do B
elif deleting: do C
The right shape:
desired = read(spec)
actual = observe(world)
diff = compare(desired, actual)
for change in diff: apply(change) # idempotently
update_status()
Reconcile must be safe to run twice in a row with no spec change → same result, zero side effects. If your function isn't idempotent, your operator will fight itself.
CRD design — the non-negotiables
crd_validator.py checks these. The list is short on purpose; each item exists because skipping it produces a specific class of bug.
| Rule | Reason it matters |
|---|---|
| Status subresource enabled | Without it, Status().Update() re-triggers the spec watcher → infinite reconcile loop |
scope: Namespaced unless cluster-scoped is justified |
Cluster-scoped CRDs leak across tenants and complicate RBAC |
served: true AND storage: true on exactly one version |
Multiple storage: true is invalid; missing it breaks reads |
| OpenAPI v3 schema with typed properties | x-kubernetes-preserve-unknown-fields: true at the root defeats validation; users send garbage, controller crashes |
conditions array in schema (for metav1.Conditions) |
Standard way to communicate state; tooling and humans both read it |
additionalPrinterColumns include Age + Status/Phase |
kubectl get becomes useful without -o yaml |
| Singular + listKind defined | kubectl get singular works |
Beyond the validator: version your CRD from day 1 (v1alpha1 → v1beta1 → v1) and plan a conversion webhook before you ship v1. Migrating storage versions later is painful.
Reconcile loop — the non-negotiables
reconcile_lint.py checks Go reconcile functions for these. Regex-based, so false positives happen — read the source after a flag.
| Rule | Why |
|---|---|
Return shape is (ctrl.Result, error) |
controller-runtime contract |
Errors return RequeueAfter or non-nil error |
Otherwise transient failures wedge silently |
Status updates use Status().Update(), not Update() |
Updating spec bumps generation; users see "ghost" changes |
No time.Sleep inside reconcile |
Blocks the workqueue → other CRs stall. Use RequeueAfter |
HTTP calls take a context.Context and respect it |
Otherwise reconciles can't be cancelled on shutdown |
| Finalizer added before any external-resource create | Missing finalizer → user deletes CR → external resource orphans |
Conditions set via meta.SetStatusCondition when CRD declares them |
Hand-rolled condition munging drops LastTransitionTime |
| Reconcile function under ~80 lines | Extract reconcileXxx subroutines per concern |
OperatorHub capability levels
operator_capability_audit.py scores against the OperatorHub 5-level rubric. Use it as a progression plan, not a checklist for one PR.
| Level | What it means | Realistic milestone |
|---|---|---|
| L1 Basic Install | CRD defined, controller deploys it | Same PR as the first reconcile |
| L2 Seamless Upgrades | PDB, conversion webhook, version skew strategy | Before first external user |
| L3 Full Lifecycle | Backup, restore, failure recovery | Before public release |
| L4 Deep Insights | /metrics endpoint, Prometheus rules, alerts |
After L3 in production for ~1 quarter |
| L5 Auto Pilot | Auto-scaling, auto-tuning, anomaly detection | Mature operators only |
Aim for L3 before public release. L4/L5 are nice-to-have unless the operator manages something high-stakes.
Framework choice
| Framework | Language | When to pick |
|---|---|---|
| kubebuilder | Go | Default for Go shops. Most opinionated scaffolding, aligned with SIGs |
| controller-runtime | Go | When kubebuilder's scaffolding is in the way (large existing codebase) |
| operator-sdk | Go / Helm / Ansible | Targeting OpenShift, or mixed-paradigm teams |
| KOPF | Python | Python shops; great for prototypes and async-heavy operators |
| metacontroller | Any (webhook) | Polyglot team that cannot pick a single language |
| java-operator-sdk | Java | JVM-only shops |
Rule of thumb: Go shop → kubebuilder. Python shop → KOPF. Build a one-week proof-of-concept before committing the design.
Workflows
Bootstrap a new operator (Go + kubebuilder)
- Pick Group/Version/Kind:
apps.example.com/v1alpha1,kind=MyApp. kubebuilder init --domain example.com --repo github.com/org/myapp-operator.kubebuilder create api --group apps --version v1alpha1 --kind MyApp.- Run
crd_validator.pyon the generated CRD. Fix every FAIL before writing controller code. - Implement reconcile — start with the simplest correct version (no finalizers, no conditions). Add complexity only when a test fails without it.
- Run
reconcile_lint.pyon the controller. - Run
operator_capability_audit.py --operator-dir .— confirm L1. - Test in a
kindcluster:kubectl apply -f config/samples/. - Add status conditions + finalizers; aim for L2 in the same PR.
Audit an existing operator
- Run all three scripts.
- Triage: FAIL → block release; WARN → file an issue.
- Record current capability level in the README.
- Plan one level advancement per quarter.
Proactive triggers
- CRD without status subresource → infinite-loop bug waiting to happen. Add
subresources: {status: {}}. time.SleepinsideReconcile→ replace withreturn ctrl.Result{RequeueAfter: X}, nil.- Multi-replica operator without leader election → split-brain. Enable leader election or scale to 1.
- Reconcile mutates
obj.Spec→ controller is fighting users. Move to status or admission webhook. - No finalizer + external resource (cloud API, DB) → orphaned resources on CR deletion. Add finalizer.
x-kubernetes-preserve-unknown-fields: trueat schema root → defeats validation. Type the schema.
Asset templates
assets/crd_template.yaml— CRD with status subresource, conditions, printer columnsassets/reconcile_skeleton.go— Go reconcile function with idempotency, finalizer, condition patterns