Kubernetes Operator
Build operators that reconcile correctly. Most operator bugs are not Kubernetes bugs — they are reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants. This skill catches them deterministically before they reach a cluster.
When to use
- Building a new Kubernetes Operator (controller for a CRD)
- Reviewing an existing operator for capability-level gaps
- Auditing a CRD spec for status/conditions/finalizer correctness
- Choosing a framework (controller-runtime / kubebuilder / operator-sdk / metacontroller / KOPF)
- Designing the API surface of a Custom Resource
- Hardening RBAC, leader election, or webhook validation
When NOT to use
- Plain Helm chart packaging → use
helm-chart-builder
- Standard kubectl operations / blue-green deploys → use
senior-devops
- General k8s security posture → use
cloud-security
- "I want to run a workload" — that's a Deployment / Job, not an operator
Core principle: an operator is a reconcile loop, not a script
observe(actual) → desired = read(spec) → diff(actual, desired) → act → update(status)
↓
requeue / done
Operators that fail are the ones that:
- Treat reconcile as imperative (do this, then this, then this) instead of declarative (make actual=desired, idempotently)
- Don't requeue transient failures
- Don't use finalizers, leaving orphan resources
- Mutate spec instead of status
- Don't use the status subresource (status updates trigger spec reconciles → loop)
- Block in reconcile (long HTTP calls, locks)
- Forget leader election → split-brain on multi-replica deploys
The 3 tools below catch each of these.
Quick start
SKILL=engineering/kubernetes-operator/skills/kubernetes-operator
# Validate a CRD design
python "$SKILL/scripts/crd_validator.py" --crd config/crd/myapp.yaml
# Lint a Go reconcile function
python "$SKILL/scripts/reconcile_lint.py" --controller controllers/myapp_controller.go
# Score against OperatorHub Capability Levels (1-5)
python "$SKILL/scripts/operator_capability_audit.py" --operator-dir .
The 3 Python tools
All stdlib-only. Run with --help.
crd_validator.py
Validates a CRD YAML against operator-pattern best practices.
python scripts/crd_validator.py --crd config/crd/myapp.yaml
python scripts/crd_validator.py --crd config/crd/ --format json
Checks:
spec.versions[*].subresources.status is set (status subresource)
spec.scope is Namespaced (not Cluster) unless explicitly justified
- Singular and listKind defined
spec.versions[*].schema.openAPIV3Schema has type definitions (no x-kubernetes-preserve-unknown-fields: true at top level)
- A version is marked
served: true AND storage: true
- Conditions array is in the schema (allows
metav1.Conditions)
- Printer columns include
Age and Status/Phase
reconcile_lint.py
Lints a Go controller reconcile function for anti-patterns.
python scripts/reconcile_lint.py --controller controllers/myapp_controller.go
Checks (regex-based heuristics):
- Returns are
(ctrl.Result, error) shape
- Errors trigger a non-zero requeue (
return ctrl.Result{Requeue: true}, err)
client.Update() on the spec object is flagged (controllers should update only status)
time.Sleep inside reconcile is flagged (use RequeueAfter)
- HTTP calls without context cancellation are flagged
- Missing
defer after a finalizer add
- No
IsConditionTrue / SetCondition calls when conditions present in CRD
- Reconcile function exceeds 80 lines (extract subroutines)
operator_capability_audit.py
Scores an operator against OperatorHub's 5 Capability Levels.
python scripts/operator_capability_audit.py --operator-dir .
Levels:
- L1 — Basic Install: CRD defined, controller deploys it
- L2 — Seamless Upgrades: PDBs, conversion webhooks, version skew strategy
- L3 — Full Lifecycle: backups, restores, failure recovery
- L4 — Deep Insights: metrics endpoint, Prometheus rules, alerts
- L5 — Auto Pilot: auto-scaling, auto-tuning, anomaly detection
Reports current level + concrete next steps to advance one level.
Tooling landscape
Pick a framework based on language and complexity. See references/tooling_landscape.md.
| Framework |
Language |
Best for |
Maintenance |
| controller-runtime |
Go |
Production-grade, low-level control |
Active (sig-api-machinery) |
| kubebuilder |
Go |
Standard scaffolding, opinionated |
Active (Kubernetes SIGs) |
| operator-sdk |
Go / Helm / Ansible |
OpenShift / mixed-paradigm teams |
Active (Red Hat) |
| metacontroller |
Any (webhook-based) |
Polyglot teams, avoiding Go |
Less active |
| KOPF |
Python |
Python shops, async-first |
Active (community) |
| java-operator-sdk |
Java |
JVM shops |
Active (Red Hat / Java SIG) |
Decision rules:
- New operator + Go shop → kubebuilder
- New operator + Python shop → KOPF
- New operator + can't pick a language → metacontroller
- OpenShift target → operator-sdk
CRD design principles
See references/crd_design.md for full detail. Quick rules:
- status is the source of truth for the controller's view of the world. Spec is what the user wants; status is what the controller observed.
- Use the status subresource. Without it, status updates re-trigger reconcile (loop).
- Use Conditions.
Ready, Reconciling, Degraded. Each carries a reason and message.
- Add finalizers. Without finalizers, deletion races the controller and orphans external resources.
- Version your CRD from day 1.
v1alpha1 → v1beta1 → v1. Plan a conversion webhook.
- Validate via OpenAPI v3 schema. Don't rely on the controller for validation that should fail at admission.
- Use
additionalPrinterColumns for kubectl get. Show Age, Phase, Ready at minimum.
- Namespace your CRDs unless they manage cluster-scoped resources.
Reconcile loop principles
See references/reconcile_loop.md for full detail. Quick rules:
- Idempotent. Reconciling the same state twice → same result, zero side effects.
- Read once, decide, act. Don't observe the world repeatedly during reconcile.
- Update status, not spec. Spec belongs to the user.
- Return errors that requeue. Use
ctrl.Result{RequeueAfter: ...} for known transient cases.
- Never block. No
time.Sleep. No long HTTP calls without context.
- Use the cache. Read via the controller's cached client; only escape the cache for a specific reason.
- Leader-elect when running >1 replica. Otherwise enable single-replica mode.
- Set OwnerReferences. Cascading deletion is the operator pattern's free gift.
Workflows
Workflow 1: Bootstrap a new operator (Go + kubebuilder)
1. Pick a Group/Version/Kind: e.g., apps.example.com/v1alpha1, kind=MyApp
2. kubebuilder init --domain example.com --repo github.com/org/myapp-operator
3. kubebuilder create api --group apps --version v1alpha1 --kind MyApp
4. Run crd_validator.py on config/crd/bases/apps.example.com_myapps.yaml
→ Fix every WARN before writing controller code
5. Implement the reconcile function (Karpathy principle 2: simplest correct version first)
6. Run reconcile_lint.py on controllers/myapp_controller.go
7. Run operator_capability_audit.py --operator-dir . — confirm L1
8. Test in a kind cluster: kubectl apply -f config/samples/
9. Add status conditions; aim for L2 in the same PR
Workflow 2: Audit an existing operator
1. Run operator_capability_audit.py --operator-dir <path>
2. Run crd_validator.py --crd config/crd/
3. Run reconcile_lint.py --controller controllers/
4. Triage findings:
- FAIL → block release; fix before next deploy
- WARN → file an issue; fix in next 30 days
5. Document current capability level in README; commit
6. Plan one capability level advancement per quarter
Workflow 3: Choose a framework
1. Identify primary language constraint (team skill)
2. Identify deployment target (vanilla k8s vs OpenShift)
3. Identify operator complexity (single CRD vs multi-CRD vs cluster-wide)
4. Cross-reference with references/tooling_landscape.md
5. Build a 1-week proof-of-concept before committing
References
references/operator_pattern.md — what an operator IS, when to use vs alternatives
references/crd_design.md — CRD design principles, versioning, conversion webhooks
references/reconcile_loop.md — reconcile patterns, error handling, idempotency
references/tooling_landscape.md — framework comparison + decision tree
Slash command
/operator-audit — Run all 3 tools on an operator repo and produce a markdown report.
Asset templates
assets/crd_template.yaml — CRD with status subresource, conditions, finalizer hint, printer columns
assets/reconcile_skeleton.go — Go controller reconcile function with idempotency, conditions, finalizers, requeue patterns
Anti-patterns
time.Sleep(30 * time.Second) inside reconcile — block other reconciles. Use RequeueAfter.
r.Client.Update(ctx, obj) to set status — use r.Status().Update(ctx, obj) instead.
- No leader election + 2+ replicas — split-brain.
- No finalizer — external resources orphan on deletion.
- CRD without status subresource — status updates trigger spec reconciles (infinite loop).
- Reconcile function > 200 lines — extract reconcileXxx subroutines per condition.
x-kubernetes-preserve-unknown-fields: true on spec root — defeats validation.
- Imperative reconcile — "if creating, do A; if updating, do B; if deleting, do C". Wrong shape. Reconcile = make actual=desired, regardless of how we got here.
Verifiable success
A team using this skill should achieve:
- 100% of new CRDs pass
crd_validator.py before merge
- All reconcile functions pass
reconcile_lint.py strict mode
- Operators reach OperatorHub Capability Level 3 (Full Lifecycle) before public release
- Mean time to fix a reconcile bug: <1 day (no infinite loops in production)
1---2name: kubernetes-operator3description: Use when building a Kubernetes Operator — custom controllers that reconcile CRD state. Triggers on "build an operator", "CRD design", "reconcile loop", "controller-runtime", "kubebuilder", "operator-sdk", "metacontroller", "KOPF", "operator capability levels", or "custom resource". Ships CRD validator, reconcile-loop linter, and OperatorHub capability auditor (all stdlib Python), 4 references on the operator pattern + CRD design + reconcile patterns + tooling landscape, and a /operator-audit slash command. NOT a generic k8s skill — specifically the Operator pattern.4context: fork5version: 2.9.06author: claude-code-skills7license: MIT8tags: [kubernetes, operator, crd, controller-runtime, kubebuilder, operator-sdk, metacontroller, kopf, reconcile, devops]9compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]10metadata:11 source: https://skillmd.com/skills/alirezarezvani/kubernetes-operator12---1314# Kubernetes Operator1516Build operators that reconcile correctly. Most operator bugs are not Kubernetes bugs — they are reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants. This skill catches them deterministically before they reach a cluster.1718## When to use1920- Building a new Kubernetes Operator (controller for a CRD)21- Reviewing an existing operator for capability-level gaps22- Auditing a CRD spec for status/conditions/finalizer correctness23- Choosing a framework (controller-runtime / kubebuilder / operator-sdk / metacontroller / KOPF)24- Designing the API surface of a Custom Resource25- Hardening RBAC, leader election, or webhook validation2627## When NOT to use2829- Plain Helm chart packaging → use `helm-chart-builder`30- Standard kubectl operations / blue-green deploys → use `senior-devops`31- General k8s security posture → use `cloud-security`32- "I want to run a workload" — that's a Deployment / Job, not an operator3334## Core principle: an operator is a reconcile loop, not a script3536```37observe(actual) → desired = read(spec) → diff(actual, desired) → act → update(status)38 ↓39 requeue / done40```4142Operators that fail are the ones that:431. Treat reconcile as imperative (do this, then this, then this) instead of declarative (make actual=desired, idempotently)442. Don't requeue transient failures453. Don't use finalizers, leaving orphan resources464. Mutate spec instead of status475. Don't use the status subresource (status updates trigger spec reconciles → loop)486. Block in reconcile (long HTTP calls, locks)497. Forget leader election → split-brain on multi-replica deploys5051The 3 tools below catch each of these.5253## Quick start5455```bash56SKILL=engineering/kubernetes-operator/skills/kubernetes-operator5758# Validate a CRD design59python "$SKILL/scripts/crd_validator.py" --crd config/crd/myapp.yaml6061# Lint a Go reconcile function62python "$SKILL/scripts/reconcile_lint.py" --controller controllers/myapp_controller.go6364# Score against OperatorHub Capability Levels (1-5)65python "$SKILL/scripts/operator_capability_audit.py" --operator-dir .66```6768## The 3 Python tools6970All stdlib-only. Run with `--help`.7172### `crd_validator.py`7374Validates a CRD YAML against operator-pattern best practices.7576```bash77python scripts/crd_validator.py --crd config/crd/myapp.yaml78python scripts/crd_validator.py --crd config/crd/ --format json79```8081**Checks:**82- `spec.versions[*].subresources.status` is set (status subresource)83- `spec.scope` is `Namespaced` (not `Cluster`) unless explicitly justified84- Singular and listKind defined85- `spec.versions[*].schema.openAPIV3Schema` has type definitions (no `x-kubernetes-preserve-unknown-fields: true` at top level)86- A version is marked `served: true` AND `storage: true`87- Conditions array is in the schema (allows `metav1.Conditions`)88- Printer columns include `Age` and `Status`/`Phase`8990### `reconcile_lint.py`9192Lints a Go controller reconcile function for anti-patterns.9394```bash95python scripts/reconcile_lint.py --controller controllers/myapp_controller.go96```9798**Checks (regex-based heuristics):**99- Returns are `(ctrl.Result, error)` shape100- Errors trigger a non-zero requeue (`return ctrl.Result{Requeue: true}, err`)101- `client.Update()` on the spec object is flagged (controllers should update only status)102- `time.Sleep` inside reconcile is flagged (use `RequeueAfter`)103- HTTP calls without context cancellation are flagged104- Missing `defer` after a finalizer add105- No `IsConditionTrue` / `SetCondition` calls when conditions present in CRD106- Reconcile function exceeds 80 lines (extract subroutines)107108### `operator_capability_audit.py`109110Scores an operator against OperatorHub's 5 Capability Levels.111112```bash113python scripts/operator_capability_audit.py --operator-dir .114```115116**Levels:**117- **L1 — Basic Install:** CRD defined, controller deploys it118- **L2 — Seamless Upgrades:** PDBs, conversion webhooks, version skew strategy119- **L3 — Full Lifecycle:** backups, restores, failure recovery120- **L4 — Deep Insights:** metrics endpoint, Prometheus rules, alerts121- **L5 — Auto Pilot:** auto-scaling, auto-tuning, anomaly detection122123Reports current level + concrete next steps to advance one level.124125## Tooling landscape126127Pick a framework based on language and complexity. See `references/tooling_landscape.md`.128129| Framework | Language | Best for | Maintenance |130|---|---|---|---|131| **controller-runtime** | Go | Production-grade, low-level control | Active (sig-api-machinery) |132| **kubebuilder** | Go | Standard scaffolding, opinionated | Active (Kubernetes SIGs) |133| **operator-sdk** | Go / Helm / Ansible | OpenShift / mixed-paradigm teams | Active (Red Hat) |134| **metacontroller** | Any (webhook-based) | Polyglot teams, avoiding Go | Less active |135| **KOPF** | Python | Python shops, async-first | Active (community) |136| **java-operator-sdk** | Java | JVM shops | Active (Red Hat / Java SIG) |137138Decision rules:139- New operator + Go shop → kubebuilder140- New operator + Python shop → KOPF141- New operator + can't pick a language → metacontroller142- OpenShift target → operator-sdk143144## CRD design principles145146See `references/crd_design.md` for full detail. Quick rules:1471481. **status is the source of truth for the controller's view of the world.** Spec is what the user wants; status is what the controller observed.1492. **Use the status subresource.** Without it, status updates re-trigger reconcile (loop).1503. **Use Conditions.** `Ready`, `Reconciling`, `Degraded`. Each carries a reason and message.1514. **Add finalizers.** Without finalizers, deletion races the controller and orphans external resources.1525. **Version your CRD from day 1.** `v1alpha1` → `v1beta1` → `v1`. Plan a conversion webhook.1536. **Validate via OpenAPI v3 schema.** Don't rely on the controller for validation that should fail at admission.1547. **Use `additionalPrinterColumns` for `kubectl get`.** Show `Age`, `Phase`, `Ready` at minimum.1558. **Namespace your CRDs unless they manage cluster-scoped resources.**156157## Reconcile loop principles158159See `references/reconcile_loop.md` for full detail. Quick rules:1601611. **Idempotent.** Reconciling the same state twice → same result, zero side effects.1622. **Read once, decide, act.** Don't observe the world repeatedly during reconcile.1633. **Update status, not spec.** Spec belongs to the user.1644. **Return errors that requeue.** Use `ctrl.Result{RequeueAfter: ...}` for known transient cases.1655. **Never block.** No `time.Sleep`. No long HTTP calls without context.1666. **Use the cache.** Read via the controller's cached client; only escape the cache for a specific reason.1677. **Leader-elect when running >1 replica.** Otherwise enable single-replica mode.1688. **Set OwnerReferences.** Cascading deletion is the operator pattern's free gift.169170## Workflows171172### Workflow 1: Bootstrap a new operator (Go + kubebuilder)173174```1751. Pick a Group/Version/Kind: e.g., apps.example.com/v1alpha1, kind=MyApp1762. kubebuilder init --domain example.com --repo github.com/org/myapp-operator1773. kubebuilder create api --group apps --version v1alpha1 --kind MyApp1784. Run crd_validator.py on config/crd/bases/apps.example.com_myapps.yaml179 → Fix every WARN before writing controller code1805. Implement the reconcile function (Karpathy principle 2: simplest correct version first)1816. Run reconcile_lint.py on controllers/myapp_controller.go1827. Run operator_capability_audit.py --operator-dir . — confirm L11838. Test in a kind cluster: kubectl apply -f config/samples/1849. Add status conditions; aim for L2 in the same PR185```186187### Workflow 2: Audit an existing operator188189```1901. Run operator_capability_audit.py --operator-dir <path>1912. Run crd_validator.py --crd config/crd/1923. Run reconcile_lint.py --controller controllers/1934. Triage findings:194 - FAIL → block release; fix before next deploy195 - WARN → file an issue; fix in next 30 days1965. Document current capability level in README; commit1976. Plan one capability level advancement per quarter198```199200### Workflow 3: Choose a framework201202```2031. Identify primary language constraint (team skill)2042. Identify deployment target (vanilla k8s vs OpenShift)2053. Identify operator complexity (single CRD vs multi-CRD vs cluster-wide)2064. Cross-reference with references/tooling_landscape.md2075. Build a 1-week proof-of-concept before committing208```209210## References211212- `references/operator_pattern.md` — what an operator IS, when to use vs alternatives213- `references/crd_design.md` — CRD design principles, versioning, conversion webhooks214- `references/reconcile_loop.md` — reconcile patterns, error handling, idempotency215- `references/tooling_landscape.md` — framework comparison + decision tree216217## Slash command218219`/operator-audit` — Run all 3 tools on an operator repo and produce a markdown report.220221## Asset templates222223- `assets/crd_template.yaml` — CRD with status subresource, conditions, finalizer hint, printer columns224- `assets/reconcile_skeleton.go` — Go controller reconcile function with idempotency, conditions, finalizers, requeue patterns225226## Anti-patterns227228- **`time.Sleep(30 * time.Second)` inside reconcile** — block other reconciles. Use `RequeueAfter`.229- **`r.Client.Update(ctx, obj)` to set status** — use `r.Status().Update(ctx, obj)` instead.230- **No leader election + 2+ replicas** — split-brain.231- **No finalizer** — external resources orphan on deletion.232- **CRD without status subresource** — status updates trigger spec reconciles (infinite loop).233- **Reconcile function > 200 lines** — extract reconcileXxx subroutines per condition.234- **`x-kubernetes-preserve-unknown-fields: true` on spec root** — defeats validation.235- **Imperative reconcile** — "if creating, do A; if updating, do B; if deleting, do C". Wrong shape. Reconcile = make actual=desired, regardless of how we got here.236237## Verifiable success238239A team using this skill should achieve:240241- 100% of new CRDs pass `crd_validator.py` before merge242- All reconcile functions pass `reconcile_lint.py` strict mode243- Operators reach OperatorHub Capability Level 3 (Full Lifecycle) before public release244- Mean time to fix a reconcile bug: <1 day (no infinite loops in production)