Build operators that reconcile correctly
An operator is a reconcile loop, not a script. Most operator bugs are not Kubernetes bugs — they are reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants.
The reconcile mental model
observe(actual) → desired = read(spec) → diff(actual, desired) → act → update(status)
↓
requeue / done
Idempotent, not imperative. The reconcile function must be callable any number of times with the same outcome. It must not assume what happened on the previous call.
Common operator bugs
| Bug |
Prevention |
| Blocking HTTP calls in reconcile |
Use async clients or move to a goroutine |
| No requeue on transient error |
Always return ctrl.Result{}, err for transient; RequeueAfter for polling |
| Missing finalizer |
Add finalizer on creation; remove only after cleanup complete |
| Mutating spec instead of status |
Spec is user-owned; status is controller-owned |
| No status subresource |
Status updates without subresource trigger spec reconcile → loop |
| RBAC over-grant |
Principle of least privilege; use ClusterRole only when namespace-scope is insufficient |
| No leader election |
Multi-replica deploy without leader election → split-brain |
Operator capability levels (OLM)
Level 1 (Basic install) → Level 2 (Seamless upgrades) → Level 3 (Full lifecycle) → Level 4 (Deep insights) → Level 5 (Auto pilot)
Start at Level 1 and promote only when the lower levels are tested and stable.
Steps
- Read the lore + existing operators.
search_lore for CRD conventions, RBAC policy, and framework choice. Match the existing toolchain (kubebuilder vs operator-sdk vs KOPF vs metacontroller).
- Design the CRD API surface.
spec = desired state (user-owned); status = observed state (controller-owned). Define conditions as standard status fields. Validate with admission webhooks for types that have invariants.
- Implement the reconcile loop. Read → diff → act → update status. Every
return is either Result{}, nil (done, no requeue) or Result{}, err / Result{RequeueAfter: d}, nil (retry/poll).
- Add finalizers. Register on object creation; execute cleanup in the finalizer block; remove only after cleanup succeeds.
- RBAC. Generate from controller-gen annotations; scope to namespace where possible; document every
ClusterRole grant.
- Leader election. Enable for any operator that will run with >1 replica.
- Verify. Deploy to a local cluster (kind/minikube); create/update/delete a CR; confirm status conditions reflect the current state; test error injection (unavailable dependency); record evidence.
Build / Test
go vet + golangci-lint clean.- Unit tests for the reconcile function with a fake client — test every
return path.
- E2E tests against a real cluster (kind): create, update (in-place upgrade), delete with finalizer cleanup.
controller-gen generates CRD + RBAC manifests — commit generated files, do not hand-edit them.
Review checklist
- Reconcile is idempotent — safe to call N times with the same result.
- Every error path requeues — transient errors return
err; expected polls use RequeueAfter.
- Finalizer present — and cleanup is tested on deletion.
- Status uses conditions — standard Kubernetes condition format (
Type, Status, Reason, Message).
- RBAC minimal — no
ClusterRole without justification; generated by controller-gen.
- Leader election enabled for multi-replica deploys.
Capture lore
Framework choice, CRD naming conventions, cluster version, and RBAC policy are high-value lore — call suggest_lore with tags: [kubernetes, operator, crd].
1---2name: kubernetes-operator3description: Use when building a Kubernetes Operator — custom controllers that reconcile CRD state. Triggers on "build an operator", "CRD design", "reconcile loop", "controller-runtime", "kubebuilder", "operator-sdk", "custom resource", or "operator capability levels". NOT a generic k8s skill — specifically the Operator pattern.4---56# Build operators that reconcile correctly78An operator is a reconcile loop, not a script. Most operator bugs are not Kubernetes bugs — they are reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants.910## The reconcile mental model1112```13observe(actual) → desired = read(spec) → diff(actual, desired) → act → update(status)14 ↓15 requeue / done16```1718**Idempotent, not imperative.** The reconcile function must be callable any number of times with the same outcome. It must not assume what happened on the previous call.1920## Common operator bugs2122| Bug | Prevention |23|-----|-----------|24| Blocking HTTP calls in reconcile | Use async clients or move to a goroutine |25| No requeue on transient error | Always `return ctrl.Result{}, err` for transient; `RequeueAfter` for polling |26| Missing finalizer | Add finalizer on creation; remove only after cleanup complete |27| Mutating spec instead of status | Spec is user-owned; status is controller-owned |28| No status subresource | Status updates without subresource trigger spec reconcile → loop |29| RBAC over-grant | Principle of least privilege; use `ClusterRole` only when namespace-scope is insufficient |30| No leader election | Multi-replica deploy without leader election → split-brain |3132## Operator capability levels (OLM)3334Level 1 (Basic install) → Level 2 (Seamless upgrades) → Level 3 (Full lifecycle) → Level 4 (Deep insights) → Level 5 (Auto pilot)3536Start at Level 1 and promote only when the lower levels are tested and stable.3738## Steps39401. **Read the lore + existing operators.** `search_lore` for CRD conventions, RBAC policy, and framework choice. Match the existing toolchain (kubebuilder vs operator-sdk vs KOPF vs metacontroller).412. **Design the CRD API surface.** `spec` = desired state (user-owned); `status` = observed state (controller-owned). Define `conditions` as standard status fields. Validate with admission webhooks for types that have invariants.423. **Implement the reconcile loop.** Read → diff → act → update status. Every `return` is either `Result{}, nil` (done, no requeue) or `Result{}, err` / `Result{RequeueAfter: d}, nil` (retry/poll).434. **Add finalizers.** Register on object creation; execute cleanup in the finalizer block; remove only after cleanup succeeds.445. **RBAC.** Generate from controller-gen annotations; scope to namespace where possible; document every `ClusterRole` grant.456. **Leader election.** Enable for any operator that will run with >1 replica.467. **Verify.** Deploy to a local cluster (kind/minikube); create/update/delete a CR; confirm status conditions reflect the current state; test error injection (unavailable dependency); record evidence.4748## Build / Test4950- `go vet` + `golangci-lint` clean.51- Unit tests for the reconcile function with a fake client — test every `return` path.52- E2E tests against a real cluster (kind): create, update (in-place upgrade), delete with finalizer cleanup.53- `controller-gen` generates CRD + RBAC manifests — commit generated files, do not hand-edit them.5455## Review checklist5657- **Reconcile is idempotent** — safe to call N times with the same result.58- **Every error path requeues** — transient errors return `err`; expected polls use `RequeueAfter`.59- **Finalizer present** — and cleanup is tested on deletion.60- **Status uses conditions** — standard Kubernetes condition format (`Type`, `Status`, `Reason`, `Message`).61- **RBAC minimal** — no `ClusterRole` without justification; generated by controller-gen.62- **Leader election enabled** for multi-replica deploys.6364## Capture lore6566Framework choice, CRD naming conventions, cluster version, and RBAC policy are high-value lore — call `suggest_lore` with `tags: [kubernetes, operator, crd]`.
tmj-90