Debugging Cockroachdb Operator Migrations

Debugging CockroachDB Operator Migrations

Debugs migration from Helm StatefulSet or public operator v1alpha1 workloads to CockroachDB Operator v1beta1 CrdbNode management. Use this after collecting the general baseline from diagnosing-cockroachdb-helm-deployments.

When to Use This Skill

• A Helm StatefulSet to operator migration is stuck or unclear
• A public operator v1alpha1 to v1beta1 migration has conversion or webhook issues
• Migration labels remain start or finalized
• status.migration.phase or status.migration.message reports an error
• Source StatefulSet ownership, pod ownerReferences, or PVC ownerReferences are unclear
• The operator stops reconciling after migration

Safety Considerations

• Do not delete the source StatefulSet until the migration phase is finalized and the customer is ready to complete migration.
• Do not modify the source StatefulSet spec during migration.
• Do not remove the migration label crdb.io/migrate during an active migration.
• Do not manually create CrdbNode resources during migration.
• Do not delete migrated PVCs. They contain CockroachDB data.
• If the operator stalls after migration, collect the escalation packet before restarting it.

Execution Discipline

• Execute one step at a time and inspect the output before moving on. Migration phase, source workload state, and ownership determine which later checks are safe.
• Do not delete the source StatefulSet, patch labels, patch mode, restart the operator, run Helm upgrades, or change ownerReferences unless the user explicitly approves the action for the target cluster.
• Do not run interactive kubectl exec shells or debug containers unless the user approves them and the customer policy allows the image/source.
• In production or when the migration phase is ambiguous, involve TSE or the operator team before changing source or target resources.

Required Inputs

• Migration type: Helm StatefulSet to operator, or public operator v1alpha1 to v1beta1
• Source resource name and namespace
• Target CrdbCluster name and namespace
• Operator namespace and version
• Values file or migration command used
• Current migration label and phase
• Whether the source StatefulSet has already been deleted

Step 1: Confirm Migration Documentation and Versions

kubectl -n <operator-namespace> get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl get crd crdbclusters.crdb.cockroachlabs.com -o jsonpath='{.spec.versions[*].name}{"\n"}'
helm -n <cockroachdb-namespace> history <cockroachdb-release> || true

Check the local migration guide and chart changelog for version-specific migration fixes:

• v1alpha1 to v1beta1 Migration Guide
• CockroachDB Helm Chart Changelog

Step 2: Inspect Migration State

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o json | jq '{
  mode: .spec.mode,
  migrateLabel: .metadata.labels["crdb.io/migrate"],
  migrationStatus: .metadata.labels["crdb.cockroachlabs.com/migration"],
  migrationPhase: .status.migration.phase,
  migrationMessage: .status.migration.message,
  initialized: [.status.conditions[]? | select(.type=="Initialized" or .type=="ClusterInitialized")],
  generation: .metadata.generation,
  observedGeneration: .status.observedGeneration
}'

kubectl -n <operator-namespace> logs -l app=cockroach-operator --tail=300 | grep -Ei 'migrationctrl|migration|phase|cert' || true
kubectl -n <cockroachdb-namespace> get crdbnodes -o wide
kubectl -n <cockroachdb-namespace> get crdbnodes -o yaml

Step 3: Inspect Source Workload

For Helm StatefulSet migration:

kubectl -n <cockroachdb-namespace> get sts <source-statefulset> -o json | jq '{
  replicas: .spec.replicas,
  readyReplicas: .status.readyReplicas,
  migrateLabel: .metadata.labels["crdb.io/migrate"],
  ownerReferences: .metadata.ownerReferences
}'

kubectl -n <cockroachdb-namespace> get sts <source-statefulset> -o yaml
kubectl -n <cockroachdb-namespace> get pods -l app.kubernetes.io/name=cockroachdb -o yaml | grep -E 'name:|ownerReferences:|kind:|uid:|controller:' -A8
kubectl -n <cockroachdb-namespace> get pvc -o yaml | grep -E 'name:|ownerReferences:|kind:|uid:|controller:' -A8

For public operator v1alpha1 migration:

kubectl -n <cockroachdb-namespace> get crdbcluster.crdb.cockroachlabs.com <cluster-name> -o yaml
kubectl -n <cockroachdb-namespace> get crdbcluster.v1beta1.crdb.cockroachlabs.com <cluster-name> -o yaml 2>&1 || true
kubectl -n <operator-namespace> get svc cockroach-webhook-service
kubectl -n <operator-namespace> get endpoints cockroach-webhook-service
kubectl get validatingwebhookconfigurations | grep cockroach

The operator supports both v1alpha1 and v1beta1 through conversion webhooks. If conversion fails, check webhook service health and operator logs.

Step 4: Interpret Migration Phases

Step 5: Debug Common Migration Stalls

Migration Not Progressing

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o jsonpath='{.status.migration}{"\n"}'
kubectl -n <operator-namespace> logs -l app=cockroach-operator --tail=300 | grep -Ei 'migration|phase|cert|error' || true
kubectl -n <operator-namespace> get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}'

Check:

• The migration feature is enabled in operator args, if the deployed version uses a feature flag.
• The source StatefulSet still exists when phase is before finalized.
• The source StatefulSet spec has not been changed mid-migration.
• Certificate prerequisites are present and trusted.
• CrdbNode objects are being created and observed.

CertMigration Issues

Use configuring-cockroachdb-helm-tls. Confirm:

• CA ConfigMap or Secret exists.
• Node, HTTP, and root client certificate resources exist.
• Certificate expiry, issuer, subject, and SANs match generated pod DNS names.
• Cert-manager Certificate resources are Ready, when cert-manager is used.
• Secret keys are present without printing private key values.

PodMigration Issues

kubectl -n <cockroachdb-namespace> get pods -l app.kubernetes.io/name=cockroachdb -o wide
kubectl -n <cockroachdb-namespace> describe pods -l app.kubernetes.io/name=cockroachdb
kubectl -n <cockroachdb-namespace> get crdbnodes -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,NODE_ID:.status.nodeID,OBSERVED:.status.observedGeneration
kubectl -n <cockroachdb-namespace> get pvc -o json | jq '.items[] | {name: .metadata.name, ownerReferences: .metadata.ownerReferences, storageClass: .spec.storageClassName, capacity: .status.capacity}'

Check:

• CrdbNode resources are created for expected pods.
• Pods and PVCs have expected ownership after the operator takes over.
• PVCs are not orphaned before deleting source resources.

Finalization or Completion Stuck

Check:

• mode is MutableOnly, not Disabled.
• migration label is finalized before deleting the source StatefulSet.
• source StatefulSet deletion happened only after finalization.
• migration label eventually becomes complete.
• operator logs show a final reconcile after the source StatefulSet delete event.

Step 6: Post-Migration Operator Stall

If the operator stops reconciling after migration completes:

1. Check the initialization condition. If missing on an already-initialized cluster, the operator may try to initialize and block.
2. Verify mode is MutableOnly.
3. Verify the migration label is complete, not start or finalized.
4. Use collecting-cockroachdb-operator-escalation-packet to collect pprof goroutine dump, metrics, and logs before restart.

Output Format

Return findings in this order:

1. Migration type and current phase
2. Source workload state
3. Target CrdbCluster mode, migration labels, and migration message
4. CrdbNode, pod, and PVC ownership state
5. Current blocker and likely cause
6. Safe next action
7. Whether escalation packet collection is required

References

• diagnosing-cockroachdb-helm-deployments
• collecting-cockroachdb-operator-escalation-packet
• v1alpha1 to v1beta1 Migration Guide
• CockroachDB Helm Chart Changelog