Collecting Cockroachdb Operator Escalation Packet

Collecting CockroachDB Operator Escalation Packet

Collects the artifacts needed for TSC/TSE or operator-team escalation. Use this after basic diagnosis identifies an unresolved operator, Kubernetes, migration, upgrade, scale, DNS, PVC, or certificate issue. Keep collection read-only unless the customer explicitly approves a mitigation.

When to Use This Skill

• A customer needs to escalate an operator-managed CockroachDB Helm issue to TSC
• The operator is running but not reconciling and logs/status do not explain why
• A rollout, upgrade, migration, or scale operation is stuck
• Operator logs do not explain the failure
• TSC needs consistent artifacts before advising restart, rollback, manual decommission, or status repair

Safety Considerations

• Collect this packet before restarting, scaling, or deleting the operator during active operations.
• Do not delete CrdbNode, PVC, Secret, service, version checker job, or StatefulSet resources while collecting data.
• Do not print private key contents. Collect only Secret names, keys, certificate metadata, expiry, issuer, subject, and SANs.
• Do not manually edit CrdbCluster.status.
• Do not run cockroach init on an existing cluster.
• If the operator is paused or disabled, record that state before changing anything.

Execution Discipline

• Execute one step at a time and inspect the output before moving on. Do not run the packet collection in parallel; earlier results determine which operation-specific sections are relevant.
• Keep collection read-only unless the user explicitly approves a mutating action for the target cluster.
• Do not run interactive kubectl exec shells, kubectl debug, port-forwards, pprof/metrics collection, or commands that use external images in production unless the user approves them. If impact or policy is unclear, involve TSE or the operator team first.
• Do not patch, annotate, delete, restart, scale, drain, decommission, or run helm upgrade as part of packet collection.

Required Inputs

• Operator namespace and Helm release
• CockroachDB namespace and Helm release
• Kubernetes context and cluster/provider
• Current operation: install, upgrade, scale up/down, cert rotation, migration, routine maintenance, or recovery
• Current and target CockroachDB image, if an upgrade is involved
• Customer timeline: what changed, what commands were run, and what was already tried

Packet Layout

Ask the customer to gather outputs into a single directory:

mkdir -p crdb-operator-escalation

Use clear filenames, for example operator-logs.txt, crdbcluster.yaml, events.txt, goroutine_dump.txt, and metrics_dump.txt.

Step 1: Context and Version Inventory

kubectl config current-context > crdb-operator-escalation/kube-context.txt
kubectl version --short > crdb-operator-escalation/kubernetes-version.txt

helm -n <operator-namespace> status <operator-release> > crdb-operator-escalation/operator-helm-status.txt 2>&1 || true
helm -n <operator-namespace> history <operator-release> > crdb-operator-escalation/operator-helm-history.txt 2>&1 || true
helm -n <cockroachdb-namespace> status <cockroachdb-release> > crdb-operator-escalation/crdb-helm-status.txt 2>&1 || true
helm -n <cockroachdb-namespace> history <cockroachdb-release> > crdb-operator-escalation/crdb-helm-history.txt 2>&1 || true

kubectl -n <operator-namespace> get deploy cockroach-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' > crdb-operator-escalation/operator-image.txt
kubectl get crd crdbclusters.crdb.cockroachlabs.com crdbnodes.crdb.cockroachlabs.com -o wide > crdb-operator-escalation/crds.txt
kubectl get crd crdbclusters.crdb.cockroachlabs.com -o yaml > crdb-operator-escalation/crdbclusters-crd.yaml
kubectl get crd crdbnodes.crdb.cockroachlabs.com -o yaml > crdb-operator-escalation/crdbnodes-crd.yaml

Get the CockroachDB version from a Ready pod:

kubectl -n <cockroachdb-namespace> exec <ready-crdb-pod> -c cockroachdb -- \
  /cockroach/cockroach version > crdb-operator-escalation/crdb-version.txt 2>&1

Step 2: Resource Specifications and Status

kubectl -n <operator-namespace> get deploy,pod,svc,endpoints -o wide > crdb-operator-escalation/operator-resources.txt
kubectl -n <operator-namespace> describe deploy cockroach-operator > crdb-operator-escalation/operator-deploy-describe.txt
kubectl -n <operator-namespace> describe pods -l app=cockroach-operator > crdb-operator-escalation/operator-pods-describe.txt
kubectl -n <operator-namespace> get deploy cockroach-operator -o yaml > crdb-operator-escalation/operator-deploy.yaml

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o yaml > crdb-operator-escalation/crdbcluster.yaml
kubectl -n <cockroachdb-namespace> describe crdbcluster <cockroachdb-release> > crdb-operator-escalation/crdbcluster-describe.txt
kubectl -n <cockroachdb-namespace> get crdbnodes -o yaml > crdb-operator-escalation/crdbnodes.yaml
kubectl -n <cockroachdb-namespace> describe crdbnodes > crdb-operator-escalation/crdbnodes-describe.txt
kubectl -n <cockroachdb-namespace> get pod,svc,endpoints,pvc,pdb -o wide > crdb-operator-escalation/crdb-resources-wide.txt
kubectl -n <cockroachdb-namespace> describe pods -l app.kubernetes.io/name=cockroachdb > crdb-operator-escalation/crdb-pods-describe.txt
kubectl -n <cockroachdb-namespace> describe pvc > crdb-operator-escalation/pvc-describe.txt
kubectl -n <cockroachdb-namespace> describe pdb > crdb-operator-escalation/pdb-describe.txt
kubectl -n <cockroachdb-namespace> get events --sort-by=.lastTimestamp > crdb-operator-escalation/events.txt

Summarize key cluster status:

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o json | jq '{
  mode: .spec.mode,
  nodes: [.spec.regions[]?.nodes],
  regions: .spec.regions,
  image: .spec.image,
  generation: .metadata.generation,
  observedGeneration: .status.observedGeneration,
  statusImage: .status.image,
  actions: .status.actions,
  conditions: .status.conditions,
  labels: .metadata.labels,
  annotations: .metadata.annotations
}' > crdb-operator-escalation/crdbcluster-summary.json

kubectl -n <cockroachdb-namespace> get crdbnodes \
  -o custom-columns=NAME:.metadata.name,GENERATION:.metadata.generation,OBSERVED:.status.observedGeneration,PHASE:.status.phase,HASH:.metadata.annotations["crdb.cockroachlabs.com/hash-revision"],NODE_ID:.status.nodeID \
  > crdb-operator-escalation/crdbnodes-summary.txt

Step 3: Logs

Collect full recent logs, not filtered snippets:

kubectl -n <operator-namespace> logs -l app=cockroach-operator --tail=500 > crdb-operator-escalation/operator-logs.txt 2>&1
kubectl -n <operator-namespace> logs -l app=cockroach-operator --previous --tail=500 > crdb-operator-escalation/operator-previous-logs.txt 2>&1 || true

For each CockroachDB pod:

kubectl -n <cockroachdb-namespace> logs <crdb-pod> -c cockroachdb --tail=500 > crdb-operator-escalation/<crdb-pod>-cockroachdb.log 2>&1
kubectl -n <cockroachdb-namespace> logs <crdb-pod> -c cockroachdb --previous --tail=500 > crdb-operator-escalation/<crdb-pod>-cockroachdb-previous.log 2>&1 || true
kubectl -n <cockroachdb-namespace> logs <crdb-pod> -c cert-reloader --tail=100 > crdb-operator-escalation/<crdb-pod>-cert-reloader.log 2>&1 || true

Step 4: Operation-Specific Evidence

Upgrade

kubectl -n <cockroachdb-namespace> get crdbcluster <cockroachdb-release> -o json | jq '{
  specImage: .spec.image,
  statusImage: .status.image,
  actions: .status.actions,
  annotations: .metadata.annotations,
  conditions: [.status.conditions[]? | select(.type | test("Upgrade|Version|Validate"))]
}' > crdb-operator-escalation/upgrade-status.json

kubectl -n <cockroachdb-namespace> get jobs -o wide > crdb-operator-escalation/jobs.txt
kubectl -n <cockroachdb-namespace> get pods -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image,PHASE:.status.phase > crdb-operator-escalation/pod-images.txt
kubectl -n <cockroachdb-namespace> describe job <version-checker-job> > crdb-operator-escalation/version-checker-job.txt 2>&1 || true
kubectl -n <cockroachdb-namespace> logs -l job-name=<version-checker-job> > crdb-operator-escalation/version-checker-logs.txt 2>&1 || true

Include current and target CRDB image and whether any version checker job or pod was deleted.

Scale Down or Decommission

kubectl -n <cockroachdb-namespace> exec <ready-crdb-pod> -c cockroachdb -- \
  /cockroach/cockroach node status --decommission > crdb-operator-escalation/node-decommission-status.txt 2>&1

kubectl -n <cockroachdb-namespace> get crdbnodes -o json | jq '[.items[] | select(.status.phase=="Decommissioning")] | {count: length, nodes: [.[].metadata.name]}' \
  > crdb-operator-escalation/decommissioning-crdbnodes.json

Include original and target node count, whether multiple nodes changed at once, and whether manual drain/decommission was attempted.

Scale Up

Capture whether machines, Kubernetes nodes, disks, storage classes, topology spread constraints, or node labels changed:

kubectl get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone > crdb-operator-escalation/nodes-locality.txt
kubectl get storageclass > crdb-operator-escalation/storageclasses.txt
kubectl -n <cockroachdb-namespace> get pvc -o wide > crdb-operator-escalation/pvc-wide.txt

Certificate Rotation

kubectl -n <cockroachdb-namespace> get secret,configmap | grep -E 'ca|node|client|tls|cert' > crdb-operator-escalation/cert-resources.txt || true
kubectl -n <cockroachdb-namespace> get certificate,issuer,clusterissuer -o wide > crdb-operator-escalation/cert-manager-resources.txt 2>&1 || true

kubectl -n <cockroachdb-namespace> get secret <node-tls-secret> -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -dates -subject -issuer -ext subjectAltName > crdb-operator-escalation/node-cert-metadata.txt

Do not collect private key values.

Migration

Use debugging-cockroachdb-operator-migrations and attach its migration state output. Include source StatefulSet or v1alpha1 CrdbCluster, migration labels, migration phase, source ownerReferences, and migration controller logs.

Step 5: Operator pprof and Metrics

Collect these when the operator is running but not reconciling, logs/status do not explain why, or a worker may be blocked. In production or restricted environments, confirm with the customer, TSE, or the operator team before opening port-forwards.

In one terminal:

kubectl -n <operator-namespace> port-forward deployment/cockroach-operator 7080:7080

In another terminal:

curl -s 'http://localhost:7080/debug/pprof/goroutine?debug=2' > crdb-operator-escalation/goroutine_dump.txt
curl -s 'http://localhost:7080/debug/pprof/goroutine?debug=1' > crdb-operator-escalation/goroutine_summary.txt
curl -s 'http://localhost:7080/debug/pprof/heap' > crdb-operator-escalation/heap.prof
curl -s 'http://localhost:7080/debug/pprof/profile?seconds=30' > crdb-operator-escalation/cpu.prof
curl -s 'http://localhost:7080/debug/pprof/mutex' > crdb-operator-escalation/mutex.prof

Metrics:

kubectl -n <operator-namespace> port-forward deployment/cockroach-operator 8080:8080
curl -s http://localhost:8080/metrics > crdb-operator-escalation/metrics_dump.txt
grep 'controller_runtime_reconcile_total' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/reconcile-total.txt || true
grep 'controller_runtime_reconcile_errors_total' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/reconcile-errors.txt || true
grep 'workqueue_depth' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/workqueue-depth.txt || true
grep 'workqueue_longest_running_processor_seconds' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/workqueue-longest-running.txt || true
grep 'workqueue_unfinished_work_seconds' crdb-operator-escalation/metrics_dump.txt > crdb-operator-escalation/workqueue-unfinished.txt || true

What TSC should inspect:

• processNextWorkItem followed by Reconcile and a long-running call in goroutine_dump.txt
• kube.(*Ctl).Exec, cockroach init, HTTP calls, or mutex waits with long durations
• increasing workqueue_longest_running_processor_seconds
• growing workqueue_depth
• no reconcile count movement for a controller that should be active

Step 6: Timeline

Create crdb-operator-escalation/timeline.md with:

• When the issue started
• What operation was in progress
• What docs or runbooks were followed
• Whether VMs, Kubernetes nodes, disks, storage classes, certs, network policies, or Helm values changed
• Every manual delete, patch, edit, restart, drain, decommission, or Helm command, in order
• What is currently serving traffic and what is unavailable
• Customer restrictions, such as no cluster-admin, no debug containers, no external images, or private registry only

Step 7: Escalation Summary

Return a concise summary with:

1. Operation type and current impact
2. Current operator and CockroachDB versions
3. Current CrdbCluster mode, generation, observedGeneration, status image, and actions
4. Stuck resource or symptom
5. Any unsafe operations already performed
6. Missing artifacts, if any
7. Whether pprof/metrics suggest a blocked worker

References

• diagnosing-cockroachdb-helm-deployments
• debugging-cockroachdb-operator-migrations
• CockroachDB Helm Chart Changelog