Rook Ceph Operator

Rook-Ceph Operator CRDs

Charts: rook-ceph + rook-ceph-cluster (from https://charts.rook.io/release) + ceph-csi-drivers (v1.20+, from https://charts.ceph.io/ceph-csi-operator)
Latest: v1.20.x (charts) | Ceph: v20.2.2 (Tentacle) | API: ceph.rook.io/v1
Install/upgrade order: rook-ceph → ceph-csi-drivers → rook-ceph-cluster

⚠️ v1.20 breaking change: CSI drivers are now managed by the separate ceph-csi-drivers Helm chart. Rook no longer creates CSI ServiceAccounts, RBAC, or Driver CRs. Without this chart, all PVC provisioning breaks. See Upgrade Notes below.

CRD Reference

Deployed Pattern

# CephCluster — via rook-ceph-cluster chart values.cephClusterSpec
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 1
  dashboard:
    enabled: true
    ssl: false
  storage:
    useAllNodes: false
    useAllDevices: false
    nodes:
      - name: worker-01
        deviceFilter: ^vdb$
      - name: worker-02
        deviceFilter: ^vdb$
        config:
          deviceClass: nvme
          osdsPerDevice: "1"
      - name: worker-03
        deviceFilter: ^vdb$
        config:
          deviceClass: nvme
          osdsPerDevice: "1"
      - name: worker-04
        deviceFilter: ^nvme
        config:
          deviceClass: nvme
          osdsPerDevice: "1"
  skipUpgradeChecks: true
  dataDirHostPath: /var/lib/rook
---
# CephBlockPool — ceph-blockpool from chart
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: ceph-blockpool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 2
  deviceClass: nvme
---
# StorageClass created automatically from CephBlockPool
# provisioner: rook-ceph.rbd.csi.ceph.com
# parameters:
#   clusterID: rook-ceph
#   imageFormat: "2"
#   imageFeatures: layering

Quick Reference

ceph-csi-drivers Chart (v1.20+)

CRITICAL: Rook v1.20 delegates CSI driver management to the ceph-csi-operator sub-chart (bundled with rook-ceph) + the separate ceph-csi-drivers chart. The drivers chart creates the ServiceAccounts, RBAC, and Driver CRs that the operator needs.

Always websearch rook/docs/v1.20/Helm-Charts/csi-drivers-chart/ for the latest recommended values.yaml before installing.

Installation

helm repo add ceph-csi-operator https://charts.ceph.io/ceph-csi-operator
helm install ceph-csi-drivers ceph-csi-operator/ceph-csi-drivers \
  --namespace rook-ceph \
  -f https://raw.githubusercontent.com/rook/rook/v1.20.2/deploy/charts/ceph-csi-drivers/values.yaml

Critical: Driver Names Must Match StorageClasses

The ceph-csi-drivers chart defaults to driver names like rbd.csi.ceph.com, but existing StorageClasses reference rook-ceph.rbd.csi.ceph.com (prefixed with the operator namespace). Set the correct names:

drivers:
  rbd:
    enabled: true
    name: rook-ceph.rbd.csi.ceph.com     # Must match StorageClass provisioner
  cephfs:
    enabled: true
    name: rook-ceph.cephfs.csi.ceph.com   # Must match StorageClass provisioner
  nfs:
    enabled: false
  nvmeof:
    enabled: false

Wrong names → StorageClass provisioner mismatch, volumes fail to provision.

Upgrade Notes (v1.19 → v1.20)

Breaking: CSI drivers moved to separate chart

In v1.19, Rook managed CSI drivers internally. In v1.20, a new ceph-csi-operator manages them via Driver CRs, and the ceph-csi-drivers Helm chart is required to create the backing ServiceAccounts + RBAC.

Installation order: rook-ceph → ceph-csi-drivers → rook-ceph-cluster

Known Issues (websearch before deploying)

1. v1.20.0: Missing SA/RBAC — Without ceph-csi-drivers chart, ctrlplugin Deployments stuck at 0/2 with serviceaccount not found. Fixed by installing the drivers chart with correct driver names. (rook#17644)

2. v1.20.1: Leader election broken — RoleBindings only included prefixed SA names, but pods use unprefixed names (rbd-ctrlplugin-sa). Provisioner can't acquire lease → PVCs hang. Fixed in v1.20.2 or patch RoleBindings to add unprefixed SAs. (rook#17787)

3. Helm upgrade conflicts — Server-side apply may conflict with existing Driver/OperatorConfig CRs managed by Rook. Add --force-conflicts or delete old CRs before upgrade.

4. Ceph v20.2.0 read-affinity bug — Use v20.2.1+ or v20.2.2.

Migration Steps

1. Upgrade rook-ceph chart to v1.20 (this deploys ceph-csi-operator sub-chart)
2. Retrieve existing CSI settings from your rook-ceph-operator-config ConfigMap
3. Install ceph-csi-drivers chart with driver names matching your StorageClasses
4. If upgrade fails with apply conflicts, use helm upgrade --force-conflicts
5. Verify: kubectl get deploy -n rook-ceph -l app.kubernetes.io/name=rbd shows ctrlplugin + nodeplugin running
6. Upgrade rook-ceph-cluster chart

Config Changes

• CSI settings removed from rook-ceph-operator-config ConfigMap — now managed via OperatorConfig + Driver CRs
• CSI custom images remain in rook-ceph chart values (csi.csiOperator.controllerManager.manager.env)
• rookUseCsiOperator: true is the new default in v1.20

Common Mistakes

• deviceFilter regex — ^vdb$ matches exactly vdb, ^nvme matches any nvme device. Wrong regex = no OSDs created.
• failureDomain: osd vs host — osd allows OSDs on same node, host spreads across nodes. Use host for production.
• CephFS dataPools[].name — Required when multiple data pools. First pool is default.
• CephFS preservePoolsOnDelete: false — Deleting the CRD deletes ALL data. Set true for safety.
• ObjectStore gateway placement — RGW must be scheduled on a node with OSDs or have network access to them.
• CephBlockPool size: 1 — No redundancy. Data lost if OSD fails. Only for test/dev or slow tier with HDD.
• OSD per device filter — osdsPerDevice: "1" creates one OSD per device. Default may create too many.
• dashboard ssl: false — Only for internal clusters. Enable SSL (ssl: true) for external access.
• Insufficient mon memory — MONS need minimum 512Mi RAM. Starving mons causes cluster instability.